Information processing device and program

Abstract

To provide an information processing apparatus and a program which analyze image data to easily generate a layout of a test performed by using a terminal device.SOLUTION: A server 10 being the information processing apparatus comprises an image data acquisition unit 31, an element extraction unit 32, an attribute recognition unit 33, a relation recognition unit 34, a structure information generation unit 35, a structure information output unit 36, an editing information acquisition unit 37, a display information acquisition unit 38, a layout generation unit 39, and a display data output unit 40. The element extraction unit 32 extracts elements of contents included in image data. The structure information generation unit 35 generates structure data relating to structures of the elements included in the image data. The structure information output unit 36 transmits the structure information of the elements included in the image data to a terminal device. The display information acquisition unit 38 acquires display information from the terminal device. The layout generation unit 39 generates a layout according to a display on the basis of the image data and the structure information.SELECTED DRAWING: Figure 3

Description

Technical Field

[0004] ,

[0006] , , ,

[0005] , , , Using a document structure recognition AI implemented with machine learning or rule-based logic, , ,

[0001] The present invention relates to a technology for reducing the workload of learning supporters.

Background Art

[0002] Conventionally, a system for conducting tests on students using a terminal device instead of paper has been known. Patent Document 1 discloses an educational support system for conducting English tests using a tablet terminal device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When conducting a test using a terminal device, it is difficult to directly use the paper-based test that has been used so far. In addition, it is necessary to generate a test layout according to the type of terminal device used by students, such as a personal computer or a general-purpose tablet PC (personal computer). Therefore, there is a problem of an increased burden on learning supporters.

[0005] The present invention has been made, for example, to solve the above problems, and an object thereof is to provide an information processing apparatus that analyzes image data and easily generates a test layout using a terminal device.

Means for Solving the Problems

[0006] In one aspect of the present invention, the information processing apparatus includes an image data acquisition unit that acquires image data from a terminal device used by a user, The test paper, including the questions and the answer spaces for those questions. and analyzes a character area and a chart area included in the image data to obtain, from the image data, Using a document structure recognition AI implemented with machine learning or rule-based logic, ​ The aforementioned Problem or The aforementioned An extraction unit that extracts the region of the element corresponding to the answer field, Using the aforementioned document structure recognition AI, An attribute recognition unit that recognizes whether the attribute of the element is the question or the answer field based on the string and diagram contained in the area of ​​the element, Using the aforementioned document structure recognition AI, A relationship recognition unit recognizes a number identifying the question and a number identifying the answer field from the string, and recognizes the correspondence between the question and the answer field as a relationship between elements based on the recognized numbers and the positional relationship of the element regions in the image data. Using the aforementioned document structure recognition AI, The system comprises a structural information generation unit that generates structural information including the regions of elements contained in the image data, the attributes of the elements, and the relationships between the elements, and a structural information output unit that transmits the structural information to the terminal device, wherein the problem consists of a major question and subordinate sub-questions, the major question is a higher-level element, the sub-question is a middle-level element, and the answer field of the sub-question is a lower-level element, and the relationship recognition unit assigns an element ID to each element obtained by sequentially combining a number identifying the major question, a number identifying the sub-question, and a number identifying the answer field of the sub-question. The structural information output unit displays, on the image data displayed by the terminal device, a rectangle enclosed by lines that associate each element with its attribute, a node corresponding to each element, and links connecting the nodes corresponding to structurally related elements. Based on the element ID, it displays a number identifying the major question on the node corresponding to the major question, a number identifying the sub-question on the link connecting the node corresponding to the major question and the node corresponding to the sub-question, and a number identifying the answer field on the link connecting the node corresponding to the sub-question and the node corresponding to the answer field.

[0007] According to this embodiment, the information processing device is By analyzing image data, the system can recognize the element's region, its attributes, and the relationships between elements, and generate structural information about the elements. Furthermore, according to this embodiment, the information processing device can acquire arbitrary image data specified by the user from a terminal device and, based on the structural information, visualize the element's region, its attributes, and the relationships between elements on the image data. In addition, even if the image data of a test paper or similar document has a structure consisting of major and minor questions, the information processing device can visualize the structure of the major questions, minor questions, and answer fields on the image data using links and numbers.

[0011] In one embodiment of the above-described information processing device, the terminal device allows the user to access the element and the attributes of the element. orThe system includes an edited information acquisition unit that acquires edited structural information in which one or more of the relationships between the elements have been edited, and the structural information generation unit replaces the edited structural information with the edited structural information. In this embodiment, the user can freely edit the element's area, the element's attributes, and the relationships between elements, and after editing, the information processing device performs predetermined processing based on the edited structural information.

[0012] In one embodiment of the above-described information processing device, Using a layout generation AI that generates layouts according to rule-based logic, The information processing device comprises a layout generation unit that generates a layout by combining the regions of elements included in the image data according to the attributes of the elements and the relationships between the elements, and a display data output unit that transmits display data for displaying the layout to the terminal device. According to this embodiment, the information processing device can freely generate a layout of content by combining elements.

[0013] In one embodiment of the above-described information processing apparatus, a resolution information acquisition unit is provided that acquires resolution information indicating the resolution of a display on an output destination terminal device that outputs the display data from the terminal device, and the layout generation unit generates the layout based on the resolution information. Depending on the resolution of the display of the output terminal device. The system generates a layout. According to this embodiment, the information processing device can generate an appropriate layout based on the resolution information and the resolution of the display of each terminal device, such as a PC or a smartphone.

[0016] In another aspect of the present invention, a program executed by an information processing device equipped with a computer is accessed from a terminal device used by a user. The test paper, including the questions and the answer spaces for those questions. Image data acquisition unit that acquires image data, Using a document structure recognition AI implemented with machine learning or rule-based logic, By analyzing the text area and chart area contained in the aforementioned image data, the image data can be used to determine: The aforementioned Problem or The aforementioned Extraction unit that extracts the region of elements corresponding to the answer field. Using the aforementioned document structure recognition AI,An attribute recognition unit that recognizes whether the attribute of the element is the question or the answer field, based on the string and diagram contained in the area of ​​the element. Using the aforementioned document structure recognition AI, A relationship recognition unit recognizes a number identifying the question and a number identifying the answer field from the string, and recognizes the correspondence between the question and the answer field as a relationship between elements based on the recognized numbers and the positional relationship of the regions of the elements in the image data. Using the aforementioned document structure recognition AI, The computer functions as a structural information generation unit that generates structural information including the regions of elements contained in the image data, the attributes of the elements, and the relationships between the elements, and a structural information output unit that transmits the structural information to the terminal device, and the problem consists of a major question and subordinate sub-questions, the major question is a higher-level element, the sub-question is a middle-level element, and the answer field of the sub-question is a lower-level element, and the relationship recognition unit assigns an element ID to each element obtained by sequentially combining a number that identifies the major question, a number that identifies the sub-question, and a number that identifies the answer field of the sub-question. The structural information output unit displays, on the image data displayed by the terminal device, a rectangle enclosed by lines that associate each element with its attribute, a node corresponding to each element, and links connecting the nodes corresponding to structurally related elements. Based on the element ID, it displays a number identifying the major question on the node corresponding to the major question, a number identifying the sub-question on the link connecting the node corresponding to the major question and the node corresponding to the sub-question, and a number identifying the answer field on the link connecting the node corresponding to the sub-question and the node corresponding to the answer field. By installing and running this program on a computer, the information processing device according to the present invention can be configured. [Effects of the Invention]

[0017] According to the information processing device of the present invention, image data can be analyzed and a test layout using a terminal device can be easily generated. [Brief explanation of the drawing]

[0018] [Figure 1]Shows the configuration of the learning support system to which the server of the present invention is applied. [Figure 2] It is a block diagram showing the hardware configuration of the server. [Figure 3] It is a block diagram showing the functional configuration of the server. [Figure 4] It is an example of image data. [Figure 5] It is an example of a rectangular area displayed on the image data. [Figure 6] It is a block diagram showing the hardware configuration of the terminal device. [Figure 7] It is an example of an editing screen. [Figure 8] It is a diagram for explaining the editing function. [Figure 9] It is an example of a layout screen. [Figure 10] It is a flowchart of the layout generation process. [Figure 11] It is an example of a link displayed on the image data.

Mode for Carrying Out the Invention

[0019] Hereinafter, embodiments of the present invention will be described with reference to the drawings. <Embodiment> [Overall Configuration] FIG. 1 shows the configuration of the learning support system to which the server of the present invention is applied. The learning support system 100 is a system that analyzes image data and recognizes the structure of elements constituting the content included in the image data. Further, the learning support system 100 is a system that generates a layout by combining elements according to the display of the output destination terminal device (hereinafter, also referred to as the "output destination terminal device"). The learning support system 100 is configured such that the server 10 and the terminal device 20 can communicate with each other via a network 5 such as the Internet.

[0020] Server 10 is an information processing device that performs processing, storage, and transmission of various types of information, and is, for example, a server device, a personal computer, or a general-purpose tablet PC.

[0021] Terminal device 20 is used by users such as teachers and test creators, and is an information processing device such as a personal computer or tablet PC. Specifically, terminal device 20 transmits image data and editing information, receives structural information and display data, and displays editing screens and layout screens.

[0022] [Server Configuration] Figure 2 is a block diagram showing the hardware configuration of server 10. Server 10 comprises a communication unit 11, a control unit 12, a storage unit 13, a recording medium 14, a display unit 15, and an input unit 16. These components are interconnected via a bus 19.

[0023] Server 10 can run on a single computer, or it can run in a distributed manner across multiple computers, or it can run in a distributed manner across virtual machines.

[0024] The communication unit 11 is a communication unit for communicating with the terminal device 20 via the network 5. Specifically, the communication unit 11 receives image data and editing information from the terminal device 20, and transmits structural information and display data to the terminal device 20.

[0025] The control unit 12 includes arithmetic processing units such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), and GPU (Graphics Processing Unit), and performs various information processing and control processing related to the server 10 by reading and executing programs stored in the memory unit 13. The programs can be deployed on a single computer or at a single site, or distributed across multiple sites and executed on multiple computers interconnected by a communication network. Although Figure 2 describes the control unit 12 as a single processor, it may also be a multi-processor system.

[0026] The storage unit 13 includes memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory), and stores programs or data necessary for the control unit 12 to execute processing. The storage unit 13 also temporarily stores data necessary for the control unit 12 to execute arithmetic processing.

[0027] The recording medium 14 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from the server 10. The recording medium 14 stores various programs that the control unit 12 executes. When the server 10 executes the layout generation process described later, the programs stored on the recording medium 14 are loaded into the storage unit 13 and executed by the control unit 12.

[0028] The display unit 15 is a liquid crystal display or an organic EL (electroluminescence) display, etc., and displays various information according to the instructions of the control unit 12. The input unit 16 is an input device such as a mouse, keyboard, touch panel, or buttons, and outputs the received operation information to the control unit 12.

[0029] Figure 3 is a block diagram showing the functional configuration of the server. Functionally, the server 10 comprises an image data acquisition unit 31, an element extraction unit 32, an attribute recognition unit 33, an association recognition unit 34, a structure information generation unit 35, a structure information output unit 36, an editing information acquisition unit 37, a display information acquisition unit 38, a layout generation unit 39, and a display data output unit 40.

[0030] The image data acquisition unit 31, element extraction unit 32, attribute recognition unit 33, relationship recognition unit 34, structure information generation unit 35, structure information output unit 36, editing information acquisition unit 37, display information acquisition unit 38, layout generation unit 39, and display data output unit 40 are realized by the control unit 12 executing a program.

[0031] Furthermore, Server 10 consists of two AI modules: a document structure recognition AI (Artificial Intelligence) and a new layout generation AI.

[0032] The document structure recognition AI is implemented using machine learning or rule-based logic. Examples of machine learning methods include deep learning such as CNN (Convolutional Neural Network). Rules for rule-based logic include, for example, based on element information, "treating elements that are close together in the image data as one content," and "the higher the position of an element in the upper left, the earlier the content containing that element is displayed, and the lower the position of an element in the lower right, the later the content containing that element is displayed," and these are set arbitrarily in advance. The document structure recognition AI analyzes the image data to extract elements from the image data and recognizes the region of the elements contained in the image data, the attributes of the elements, and the relationships between elements. The element extraction unit 32, attribute recognition unit 33, relationship recognition unit 34, and structure information generation unit 35, which will be described later, are realized by the document structure recognition AI.

[0033] The new layout generation AI generates a content layout by combining elements according to the performance and size of the display of the output terminal device that displays the layout. The new layout generation AI is implemented using rule-based logic. The layout generation unit 39, described later, is realized by the new layout generation AI.

[0034] The image data acquisition unit 31 acquires image data of teaching materials. Teaching materials include, for example, test papers containing questions and answer spaces for those questions. Specifically, the user acquires image data by scanning a paper test paper using a scanner and transmits the image data to the server 10 using the terminal device 20. The image data acquisition unit 31 receives the image data of the test paper from the terminal device 20. Note that the image data is not limited to images obtained by scanning paper teaching materials, but may also be images that were originally created as electronic data.

[0035] Figure 4 shows an example of image data. The user uses the terminal device 20 to perform a predetermined operation to acquire and display image data 60 of a test question sheet, as shown in Figure 4. The terminal device 20 then sends the image data 60 and an element generation request to the server 10.

[0036] In this embodiment, the terminal device 20 transmits the image data and the element generation request to the server 10 upon acquiring the image data. However, the present invention is not limited to this, and the image data may be transmitted in advance, with the element generation request transmitted at a different time.

[0037] Furthermore, in this embodiment, content refers to a single subject that has been divided into the smallest possible units that cannot be further subdivided. For example, the content of a test question paper consists of one question and an answer space for that question. Here, one question is a set of one main question and one sub-question belonging to that main question. An element is a component that makes up the content; for example, the main question, sub-question, and answer space of the test question paper are each one element. In other words, in the test question paper of this embodiment, one piece of content is made up by combining the three elements: the main question, the sub-question, and the answer space. Note that the structure of a test question paper varies depending on the subject, and various structures are possible, such as those without a main question or those where the sub-question and answer space are combined. Therefore, the elements that make up the content are not limited to this and can be set arbitrarily.

[0038] The element extraction unit 32 analyzes the image data and extracts the content elements contained in the image data. In this embodiment, the content elements are major questions, minor questions, and answer fields, so the element extraction unit 32 analyzes the image data and defines the area of ​​each element with rectangles. Figure 5 is an example showing rectangular areas that define the area of ​​each element contained in the image data 60. As shown in Figure 5, the image data 60 has rectangular areas 61 to 72.

[0039] The attribute recognition unit 33 recognizes the attributes of the elements extracted by the element extraction unit 32. In this embodiment, the attributes are one of the following: major question, minor question, or answer field. As shown in Figure 5, the rectangular area is surrounded by a line corresponding to the attribute; specifically, if the attribute is "major question," it is surrounded by a thick solid line; if the attribute is "minor question," it is surrounded by a thin solid line; and if the attribute is "answer field," it is surrounded by a dotted line.

[0040] For example, rectangular area 61 is surrounded by a thick solid line because the attribute recognition unit 33 recognizes its attribute as "major question". Rectangular area 62 is surrounded by a thin solid line because the attribute recognition unit 33 recognizes its attribute as "sub-question". Rectangular area 63 is surrounded by a dotted line because the attribute recognition unit 33 recognizes its attribute as "answer field". Similarly, rectangular areas 64 to 72 are surrounded by lines corresponding to the attributes recognized by the attribute recognition unit 33.

[0041] The association recognition unit 34 recognizes the relationships between the elements extracted by the element extraction unit 32. Specifically, the association recognition unit 34 assigns an element ID to each element that represents the relationship between the recognized elements. The element ID is information necessary to display the links described later, and is assigned to each element of the main question, sub-question, and answer field. The elements, namely the main question, sub-question, and answer field, have a hierarchical structure, with the main question being a higher-level element, the sub-question a middle-level element, and the answer field a lower-level element. Therefore, for example, if the element ID "1" is assigned to main question 1 in the image data 60, then the element ID "1-1" is assigned to sub-question 1 of main question 1, and the element ID "1-1-1" is assigned to the answer field of sub-question 1. Similarly, the element ID "1-2" is assigned to sub-question 2 of main question 1, and the element ID "1-2-1" is assigned to the answer field of sub-question 2. In this case, the elements of the main question contain information about the element ID of its child sub-question and the element ID of the answer field, which is a child of that sub-question. Similarly, the elements of a sub-question contain information about the element ID of its parent main question and the element ID of its child answer field. Furthermore, the elements of an answer field contain information about the element ID of its parent sub-question and the element ID of the main question that is the parent of that sub-question. Thus, the element IDs represent the relationships between elements, indicating that a single piece of content is constructed by combining main question 1, sub-question 1 of main question 1, and the answer field of sub-question 1.

[0042] Specifically, the association recognition unit 34 assigns an order representing the arrangement of the questions based on the positional relationship of the rectangular areas on the image data, using the coordinate values ​​of the rectangular areas. In other words, the association recognition unit 34 assigns an order representing the arrangement of the questions to each element constituting the questions. While the association recognition unit 34 assigns an order to each element based on the positional relationship of the rectangular areas, the present invention is not limited to this. For example, OCR (Optical Character Recognition) may be used to recognize the question numbers written in the rectangular areas of the main questions, sub-questions, and answer fields, and an order may be assigned to each element based on the positional relationship and the question number. Thus, the rules for assigning an order to each element can be set arbitrarily.

[0043] The structural information generation unit 35 generates structural information relating to the structure of elements contained in the image data. The structural information includes coordinate values ​​indicating the rectangular area of ​​the element on the image data, element attributes, and information regarding the relationships between elements. Specifically, the information regarding the relationships between elements includes element IDs representing the relationships between elements and information regarding the order in which the problems are arranged. Furthermore, when the editing information acquisition unit 37, described later, acquires editing information, the structural information generation unit 35 replaces the structural information of the elements contained in the image data 60 with the structural information after editing by the user.

[0044] The structural information output unit 36 ​​transmits structural information of the elements included in the image data to the terminal device 20. Based on the structural information received from the server 10, the terminal device 20 displays an editing screen, which will be described later. The user uses the terminal device 20 to edit the elements from the editing screen. Based on the coordinate values ​​of the rectangular area of ​​the elements edited by the user, the attributes of the elements, and the relationships between elements, the terminal device 20 generates edited structural information and transmits it to the server 10 as editing information.

[0045] The editing information acquisition unit 37 acquires the edited structure information as editing information from the terminal device 20. The display information acquisition unit 38 acquires display information from the terminal device 20 regarding the display of the output destination terminal device, which will be described later. The display of the output destination terminal device is one of the output devices of an information processing device, and is a display device that projects an image onto a screen. The display information is information about the performance and size of the display, and specifically includes information about the terminal type of the output destination terminal device, such as a tablet PC or smartphone, and information about the resolution of the display of the output destination terminal device. Note that the editing information and the display information may be acquired at different times, or the editing information may be acquired at the same time as the display information is acquired.

[0046] The layout generation unit 39 first identifies the performance and size of the display of the output terminal device based on the display information. Then, the layout generation unit 39 generates a layout corresponding to the display of the output terminal device based on the image data and structural information. For example, the layout generation unit 39 generates a test layout corresponding to the display of the output terminal device based on the image data 60 shown in Figure 4 and structural information.

[0047] The display data output unit 40 transmits display data to the terminal device 20 to display the generated layout. The terminal device 20 receives the display data and displays the layout screen, which will be described later.

[0048] In the above configuration, the image data acquisition unit 31, element extraction unit 32, attribute recognition unit 33, association recognition unit 34, structure information generation unit 35, structure information output unit 36, editing information acquisition unit 37, display information acquisition unit 38, layout generation unit 39, and display data output unit 40 of the server 10 are examples of the image data acquisition unit, extraction unit, attribute recognition unit, association recognition unit, structure information generation unit, structure information output unit, editing information acquisition unit, display information acquisition unit, layout generation unit, and display data output unit of the present invention, respectively.

[0049] [Terminal device configuration] Figure 6 is a block diagram showing the hardware configuration of the terminal device 20. The terminal device 20 comprises a communication unit 21, a control unit 22, a storage unit 23, a display unit 25, and an input unit 26. These components are interconnected via a bus 29.

[0050] The communication unit 21 is a communication unit for communicating with the server 10 via the network 5. Specifically, the communication unit 21 sends image data and editing information to the server 10, and receives structural information and display data from the server 10.

[0051] The control unit 22 includes a processing unit such as a CPU, MPU, and GPU, and performs various information processing and control processing related to the terminal device 20 by reading and executing programs stored in the storage unit 23. In Figure 6, the control unit 22 is described as a single processor, but it may be a multiprocessor.

[0052] The storage unit 23 includes memory elements such as RAM and ROM, and stores programs or data necessary for the control unit 22 to execute processing. The storage unit 23 also temporarily stores data necessary for the control unit 22 to execute arithmetic processing.

[0053] The display unit 25 is a liquid crystal display or an organic EL display, etc., and displays various information according to the instructions of the control unit 22. The input unit 26 is an input device such as a mouse, keyboard, touch panel, or buttons, and outputs the received operation information to the control unit 22.

[0054] [Edit screen] Figure 7 shows an example of an editing screen. The editing screen is displayed by the terminal device 20 based on the structural information and image data received from the server 10. The user can edit the element's area, element attributes, and relationships between elements using the editing screen. As shown in Figure 7, the editing screen has an image display area 56 that visualizes the element's structure on the left side of the UI (User Interface), and an element display area 58 that visualizes information about each element on the right side of the UI.

[0055] On the left side of the editing screen are the image loading button 51, the element generation button 52, the tablet view acquisition button 53, the smartphone view acquisition button 54, the PC view acquisition button 55, and the image display area 56.

[0056] The image loading button 51 is pressed when the user loads a predetermined image data. Upon pressing the image loading button 51, the terminal device 20 loads the image data and sends it to the server 10.

[0057] The element generation button 52 is pressed when the user requests element generation. When the element generation button 52 is pressed, the terminal device 20 sends an element generation request to the server 10. The terminal device 20 then receives structural information from the server 10.

[0058] The tablet view acquisition button 53 is pressed when the user simulates the test layout to be displayed on the tablet PC. When the tablet view acquisition button 53 is pressed, the terminal device 20 sends display information to the server 10 indicating that the terminal type of the output terminal device is a tablet PC. The terminal device 20 then receives display data from the server 10 that displays a layout corresponding to the display of the tablet PC.

[0059] The smartphone view acquisition button 54 is pressed when the user simulates the layout to be displayed on their smartphone. When the smartphone view acquisition button 54 is pressed, the terminal device 20 sends display information to the server 10 indicating that the terminal type of the output terminal device is a smartphone. The terminal device 20 then receives display data from the server 10 that displays a layout corresponding to the display of the smartphone.

[0060] The PC View Acquisition button 55 is pressed when the user simulates the layout to be displayed on the PC. When the PC View Acquisition button 55 is pressed, the terminal device 20 sends display information to the server 10 indicating that the terminal type of the output terminal device is a PC. The terminal device 20 then receives display data from the server 10 that displays a layout corresponding to the display of the PC.

[0061] Specifically, the image data is the image data 60 of the test question paper shown in Figure 4. When the image loading button 51 is pressed, the terminal device 20 displays the loaded image data 60 in the image display area 56. Furthermore, when the element generation button 52 is pressed, the terminal device 20 displays rectangular areas 61 to 72 on the image data 60, with the element areas enclosed by lines corresponding to the attributes, based on the structural information received from the server 10, as shown in Figure 7.

[0062] Each of the rectangular regions 61 to 72 has a node, and the terminal device 20 displays links between nodes with lines based on element IDs, connecting the nodes to each other. For example, as shown in Figure 7, the terminal device 20 displays a link between node 73 in rectangular region 61 of major question 1 and node 74 in rectangular region 62 of sub-question 1, which is part of major question 1. Furthermore, the terminal device 20 displays a link between node 74 in rectangular region 62 of sub-question 1 and node 75 in rectangular region 63 of the answer field for sub-question 1.

[0063] At this time, the terminal device 20 may display the element IDs included in the structural information in association with the links. For example, as shown in Figure 7, the terminal device 20 displays "1" on node 73 of the rectangular area 61 because the element ID of major question 1 is "1". Then, the terminal device 20 displays "1" on the link connecting node 73 and node 74 because the element ID of sub-question 1 is "1-1". Furthermore, the terminal device 20 displays "1" on the link connecting node 74 and node 75 because the element ID of the answer field is "1-1-1".

[0064] In this way, the elements branch out and are linked from the main question to the sub-questions, and from the sub-questions to the answer fields. By displaying the links and element IDs in image display area 56, it is possible to visualize that the main question, sub-questions, and answer fields have a hierarchical structure and that the three elements constitute one piece of content.

[0065] On the right side of the editing screen are the Add Link button 41, the Delete Link button 42, the Rectangle Size and Position buttons 43, the Add Rectangle button 44, the Delete Rectangle button 45, the Rectangle Attribute Change button 46, and the Element Display Area 58.

[0066] The element display area 58 has multiple element items, each displaying information about each element included in the image data displayed in the image display area 56. For example, as shown in Figure 7, the terminal device 20 displays the image data of the element defined by the rectangular area 61 and the element ID 85 assigned to that element in element item 81. It also displays the image data of the element defined by the rectangular area 62 and the element ID 86 assigned to that element in element item 82. Furthermore, it displays the image data of the element defined by the rectangular area 63 and the element ID 87 assigned to that element in element item 83.

[0067] Furthermore, the element display area 58 may arrange element items to visualize the relationships between elements. For example, as shown in Figure 7, the element display area 58 displays element item 81 of the higher-level main question 1 in color at the top, based on the element ID, and displays element item 82 of the element of sub-question 1, which is at the middle level, and element item 83 of the lower-level answer field, which is at the bottom of element item 81. By arranging the element items in this way, users can easily visually confirm that sub-question 1 and its answer field, sub-question 2 and its answer field, and sub-question 3 and its answer field exist at levels below main question 1. In addition, the element display area 58 may display sub-questions and their answer fields that make up questions that appear earlier in the sequence at the top, and sub-questions and their answer fields that make up questions that appear later in the sequence at the bottom. By arranging the element items in this way, users can easily confirm the order of the questions in the image data. Such arrangement of element items can be set arbitrarily.

[0068] The Add Link button 41 is pressed when the user wants to add a link between elements in the image display area 56. Pressing the Add Link button 41 causes the terminal device 20 to execute the Add Link function. Figure 8 is a diagram illustrating the element editing function in the editing screen. As shown in Figure 8(A), the Add Link function is a function that adds a link indicating the relationship between elements by having the user perform a predetermined operation in the image display area 56.

[0069] The link deletion button 42 is pressed when the user wants to delete the link between elements in the image display area 56. When the link deletion button 42 is pressed, the terminal device 20 executes the link deletion function. As shown in Figure 8(B), the link deletion function deletes the link indicating the relationship between elements by having the user perform a predetermined operation in the image display area 56.

[0070] The rectangle size and position button 43 is pressed when the user wants to modify the size and position of a rectangular area in the image display area 56. Pressing the rectangle size and position button 43 causes the terminal device 20 to execute the rectangle size and position editing function. As shown in Figure 8(C), the rectangle size and position editing function allows the user to modify the size and position of a rectangular area in the image data by performing predetermined operations in the image display area 56.

[0071] The rectangle add button 44 is pressed when the user wants to add a rectangular area in the image display area 56. When the rectangle add button 44 is pressed, the terminal device 20 executes the rectangle add function. As shown in Figure 8(D), the rectangle add function is a function that adds a new rectangular area on the image data and defines the area of ​​a new element by having the user perform a predetermined operation in the image display area 56.

[0072] The rectangle delete button 45 is pressed when the user wants to delete a rectangular area in the image display area 56. When the rectangle delete button 45 is pressed, the terminal device 20 executes the rectangle delete function. As shown in Figure 8(E), the rectangle delete function deletes a rectangular area displayed on the image data by performing a predetermined operation in the image display area 56, thereby deleting the elements defined by that rectangular area.

[0073] The rectangle attribute change button 46 is pressed when the user wants to change the attributes of an element in the image display area 56. When the rectangle attribute change button 46 is pressed, the terminal device 20 executes the rectangle attribute change function. As shown in Figure 8(F), the rectangle attribute change function is a function that changes the attributes of an element by having the user perform a predetermined operation in the image display area 56.

[0074] The coordinate values ​​of the rectangular area of ​​an element edited by the editing function, the attributes of the element, and the relationships between elements are all reflected in the element display area 58. In other words, the terminal device 20 edits the structural information based on the user's edits and displays the editing screen, including the element display area 58, based on the edited structural information.

[0075] This editing function allows users to make appropriate corrections if the element regions, element attributes, and relationships between elements predicted by various AIs are incorrect. However, the editing function is not limited to this; it can also be configured as needed, for example, by adding a function to change the order in which the problems assigned to each element are displayed.

[0076] [Layout screen] Figure 9 shows an example of a layout screen. The layout screen is a screen displayed by the terminal device 20 based on the display data received from the server 10. The user can use the layout screen to confirm a predetermined layout corresponding to the display of the output terminal device.

[0077] Specifically, the user sends display information to the server 10 by pressing the tablet view acquisition button 53, smartphone view acquisition button 54, or PC view acquisition button 55 using the terminal device 20. The terminal device 20 can then receive display data from the server 10 that displays a predetermined layout corresponding to the display of the output terminal device.

[0078] As shown in Figure 9, the layout screen includes a screen image 91, page items 94, a back button 95, and a next button 96.

[0079] Screen image 91 displays an image of the screen of the output terminal device. For example, when a user presses the tablet view acquisition button 53 to send display information to the server 10, screen image 91 displays an image of the tablet PC screen, as shown in Figure 9. Screen image 91 has a problem image 92 on which the problem is displayed and an answer field image 93 on which the answer field is displayed. When the terminal type of the output terminal device is a tablet PC or a smartphone, it is appropriate to display one problem per page. Therefore, as shown in Figure 9, problem image 92 displays one problem combining main question 1 and sub-question 2 of main question 1, and answer field image 93 displays the answer field for sub-question 2 of main question 1. In other words, screen image 91 displays a main question, sub-question, and answer field that constitute one piece of content on one page in any arrangement.

[0080] Page item 94 displays information about the number of pages. Page item 94 has a total number of pages 94a, which indicates the total number of pages, and a current page number 94b, which indicates the number of pages currently displayed in the screen image 91.

[0081] For example, the image data 60 of the editing screen shown in Figure 7 has five questions: "Sub-question 1 of main question 1", "Sub-question 2 of main question 1", "Sub-question 3 of main question 1", "Sub-question 1 of main question 2", and "Sub-question 2 of main question 1". If the layout displays one question per page, the layout generation unit 39 of the server 10 generates a total of five pages as the test layout based on the image data 60. The terminal device 20 receives display data to display the layout from the server 10 and displays the layout screen. In this case, as shown in Figure 9, the total number of pages 94a displays "5", indicating a total of five pages, and the current page number 94b displays "2", indicating the page number of the second question.

[0082] The back button 95 is a button that moves the screen to the previous page displayed in screen image 91. The next button 96 is a button that moves the screen to the next page displayed in screen image 91. This allows the user to view all pages.

[0083] Furthermore, if the output terminal device is a PC, it may display multiple problems on a single page because its display is larger than that of a tablet PC or smartphone. Thus, the number of problems displayed on a single page is arbitrarily set in advance according to the performance and size of the display of the output terminal device, and the layout generation unit 39 of the server 10 generates an appropriate layout based on the setting.

[0084] [Layout generation process] Next, we will describe the layout generation process, which recognizes the structure of elements by analyzing image data and generates a predetermined layout according to the display of the output terminal device. Figure 10 is a flowchart of the layout generation process. This process is realized by the server 10 executing a pre-prepared program.

[0085] The user uses terminal device 20 to send image data read using a scanner and an element generation request to server 10 (step S101). Server 10 analyzes the image data received from terminal device 20 and extracts the elements contained in the image data (step S102). Server 10 then recognizes the attributes of the extracted elements (step S103). Furthermore, server 10 recognizes the relationships between the extracted elements (step S104). Specifically, server 10 assigns an element ID and order to each element based on the recognized relationships between the elements. Based on the region of the extracted elements, the attributes of the elements, and the relationships between the elements, server 10 generates structural information about the structure of the elements contained in the image data and sends it to terminal device 20 (step S105).

[0086] The terminal device 20 displays an editing screen based on the structural information received from the server 10 (step S106). The terminal device 20 then determines whether or not the user has edited an element (step S107). Specifically, the terminal device 20 determines whether or not the user has pressed any of the buttons corresponding to the editing function. If it is determined that no editing has been performed (step S107; No), the terminal device 20 proceeds to step S109. On the other hand, if it is determined that editing has been performed (step S107; Yes), the terminal device 20 generates edited structural information based on the area of ​​the element edited by the user, the attributes of the element, and the relationships between elements (step S108).

[0087] The terminal device 20 sends display information to the server 10 to confirm a predetermined layout when displayed on a predetermined output terminal device (step S109). At this time, if edited structure information has been generated, the terminal device 20 sends the edited information to the server 10 along with the display information. The server 10 receives the display information from the terminal device 20 (step S110). At this time, if the server 10 has received the edited information along with the display information, the server 10 replaces the structure information of the elements included in the image data with the edited structure information. Based on the display information, the server 10 identifies the performance and size of the display of the output terminal device, and generates a predetermined layout corresponding to the display of the output terminal device based on the image data and structure information. Then, the server 10 sends display data that displays the generated layout to the terminal device 20 (step S111).

[0088] The terminal device 20 displays a predetermined layout on the layout screen based on the received display data, corresponding to the display of the output terminal device (step S112). This completes the layout generation process.

[0089] The learning support system 100 of this embodiment can visualize the regions of elements, the attributes of elements, and the relationships between elements contained in the image data by analyzing the image data and generating structural information. This allows the user to easily check and edit the structure of elements. Furthermore, since the learning support system 100 can divide and process image data into element units, it can easily generate a predetermined layout corresponding to the display of the output terminal device. For example, the learning support system 100 can easily generate a test layout corresponding to the display of the output terminal device from a paper test question sheet and provide it to the user. In this way, the learning support system 100 can provide the user with a UI that visualizes the structure of elements and a UI that allows for easy confirmation of the layout.

[0090] <Variation> Next, we will describe some modifications. The following modifications can be combined as appropriate and applied to the embodiments.

[0091] (First variation) In the above embodiment, links are represented by simple solid lines, but the present invention is not limited thereto, and links may be changed according to the relationships between elements. Figure 11 shows an example in which links are changed according to the relationships between elements. As shown in Figure 11, by making the links connecting higher-level elements and middle-level elements thicker solid lines, and the links connecting middle-level elements and lower-level elements thinner solid lines, the hierarchical structure of elements can be visualized by the thickness of the links. Alternatively, instead of simple lines, links may be represented by arrows to visualize the hierarchical structure of elements.

[0092] (Second variation) In the above embodiment, the user uses terminal device 20, but the present invention is not limited thereto, and the user may use terminal device 20x which has the functions of server 10. Terminal device 20x is, like server 10, for example, a personal computer or a general-purpose tablet PC (personal computer).

[0093] In this case, the terminal device 20x also executes the layout generation process (steps S102-105, S110, and S111) that was performed by the server 10, and can analyze image data to generate structural information and layout. The terminal device 20x is an example of the information processing apparatus of the present invention. [Explanation of Symbols]

[0094] 5 Network 10 servers 20, 20x terminal devices 31 Image data acquisition unit 32 Element Extraction Unit 33 Attribute recognition part 34 Related Recognition Unit 35 Structure information generation section 36. Structural Information Output Unit 37 Editorial Information Acquisition Department 38 Display Information Acquisition Unit 39 Layout generation unit 40 Display data output section 100 Learning Support Systems

Claims

1. An image data acquisition unit that acquires image data of the test paper, including the questions and the answer fields for those questions, from a terminal device used by the user, An extraction unit that uses a document structure recognition AI implemented with machine learning or rule-based logic to analyze the text regions and diagram regions contained in the image data, thereby extracting the regions of elements corresponding to the question or the answer field from the image data. An attribute recognition unit that uses the document structure recognition AI to recognize whether the attribute of the element is the question or the answer field based on the string and diagram contained in the element's area, A relationship recognition unit that uses the document structure recognition AI to recognize a number identifying the question and a number identifying the answer field from the string, and recognizes the correspondence between the question and the answer field as a relationship between elements based on the recognized numbers and the positional relationship of the element regions in the image data, A structural information generation unit that uses the document structure recognition AI to generate structural information including the regions of elements contained in the image data, the attributes of the elements, and the relationships between the elements, It comprises a structural information output unit that transmits the structural information to the terminal device, The aforementioned problem consists of a major question and subordinate sub-questions, where the major question is a higher-level element, the sub-questions are middle-level elements, and the answer fields for the sub-questions are lower-level elements. The association recognition unit assigns an element ID to each element, which is obtained by sequentially combining a number that identifies the main question, a number that identifies the sub-question, and a number that identifies the answer field of the sub-question. The structural information output unit displays, on the image data displayed by the terminal device, rectangles enclosed by lines that associate each element with its attribute, nodes corresponding to each element, and links connecting the nodes corresponding to structurally related elements. An information processing device that, based on the element ID, displays a number identifying the major question on the node corresponding to the major question, a number identifying the sub-question on the link connecting the node corresponding to the major question and the node corresponding to the sub-question, and a number identifying the answer field on the link connecting the node corresponding to the sub-question and the node corresponding to the answer field.

2. The terminal device includes an edited information acquisition unit that acquires structural information from the user, in which one or more of the elements, attributes of the elements, or relationships between the elements have been edited. The information processing apparatus according to claim 1, wherein the structural information generation unit replaces the structural information with edited structural information.

3. A layout generation unit that uses a layout generation AI to generate a layout according to rule-based logic, and generates a layout by combining the regions of elements included in the image data according to the attributes of the elements and the relationships between the elements, A display data output unit that transmits display data for displaying the aforementioned layout to the terminal device, The information processing apparatus according to claim 1 or 2, comprising:

4. The terminal device includes a resolution information acquisition unit that acquires resolution information indicating the resolution of the display of the output destination terminal device that outputs the display data, The information processing apparatus according to claim 3, wherein the layout generation unit generates a layout corresponding to the resolution of the display of the output destination terminal device based on the resolution information.

5. A program executed by an information processing device equipped with a computer, Image data acquisition unit that acquires image data of the test paper, including the questions and the answer fields for those questions, from the terminal device used by the user. An extraction unit that uses a document structure recognition AI implemented with machine learning or rule-based logic to analyze the text regions and diagram regions contained in the image data, thereby extracting the regions of elements corresponding to the question or the answer field from the image data. An attribute recognition unit that uses the document structure recognition AI to recognize whether the attribute of the element is the question or the answer field, based on the string and diagram contained in the element's area. A relationship recognition unit that uses the document structure recognition AI to recognize a number that identifies the question and a number that identifies the answer field from the string, and recognizes the correspondence between the question and the answer field as a relationship between elements based on the recognized numbers and the positional relationship of the element regions in the image data. A structural information generation unit generates structural information including the regions of elements contained in the image data, the attributes of the elements, and the relationships between the elements, using the aforementioned document structure recognition AI. The computer functions as a structural information output unit that transmits the aforementioned structural information to the terminal device. The aforementioned problem consists of a major question and subordinate sub-questions, where the major question is a higher-level element, the sub-questions are middle-level elements, and the answer fields for the sub-questions are lower-level elements. The association recognition unit assigns an element ID to each element, which is obtained by sequentially combining a number that identifies the main question, a number that identifies the sub-question, and a number that identifies the answer field of the sub-question. The structural information output unit displays, on the image data displayed by the terminal device, rectangles enclosed by lines that associate each element with its attribute, nodes corresponding to each element, and links connecting the nodes corresponding to structurally related elements. A program that, based on the element ID, displays a number identifying the major question on the node corresponding to the major question, a number identifying the sub-question on the link connecting the node corresponding to the major question and the node corresponding to the sub-question, and a number identifying the answer field on the link connecting the node corresponding to the sub-question and the node corresponding to the answer field.

Images