Information processing device and information processing system

Abstract

To provide an information processing apparatus configured to automatically obtain regions of a micro-content constituting a content, and an information processing system.SOLUTION: A procedure processed by an information processing apparatus includes: acquiring a content image including a content; inputting the acquired content image to a learning model for estimating regions of micro-contents constituting a content, to estimate regions of micro-contents; and displaying the estimated regions of the micro-content on the content in accordance with an estimation result in different modes.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing system. [Background technology]

[0002] In recent years, technologies for reconstructing content have become known. For example, Patent Document 1 discloses a content reconstructing device that performs structural analysis on structured content data to be reconstructed, acquires structural information on microcontents that are constituent elements of the content data to be reconstructed, and inputs the structural information of the content data to be reconstructed to automatically determine a layout to be applied to the content data to be reconstructed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-004067 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the invention of Patent Document 1 has a problem in that it is not possible to extract (obtain) from content the area of ​​microcontents that make up that content.

[0005] In one aspect, an object of the present invention is to provide an information processing device or the like that can automatically acquire the area of ​​microcontent that constitutes content. [Means for solving the problem]

[0006] An information processing device according to one aspect includes an acquisition unit that acquires a content image including content; an estimation unit that, when a content image is input, inputs the content image acquired by the acquisition unit into a learning model that estimates an area of ​​micro content constituting the content, thereby estimating the area of ​​the micro content; and a second reception unit that receives a setting for a threshold for estimation accuracy of the area of ​​the micro content. a third receiving unit that receives a setting for a threshold value of the degree of overlap of the microcontent areas; and a first extracting unit that extracts a microcontent area where the estimation accuracy of the microcontent area estimated by the estimation unit is equal to or less than the threshold value of the estimation accuracy received by the second receiving unit; a display unit that displays the estimated area of ​​each micro content on the content in a different manner according to the estimation result obtained by the estimation unit; the display unit displays the micro content areas on the content for each type of the micro content area, displays the micro content areas extracted by the first extraction unit on the content, and when the degree of overlap of the micro content areas estimated by the estimation unit is equal to or greater than the threshold value of the degree of overlap accepted by the third acceptance unit, displays the areas in different colors corresponding to the degree of overlap. . [Effects of the Invention]

[0007] In one aspect, it is possible to automatically acquire the area of ​​microcontents that constitute the content. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram showing an overview of a content extraction system for micro content production. [Figure 2] FIG. 1 is a block diagram illustrating an example of the configuration of a computer. [Figure 3] FIG. 2 is a block diagram illustrating an example of the configuration of a server. [Figure 4] FIG. 10 is an explanatory diagram showing an example of a record layout of a micro content DB. [Figure 5] FIG. 10 is an explanatory diagram illustrating a region estimation model. [Figure 6] 10 is a flowchart showing a processing procedure for estimating a region of microcontent. [Figure 7] FIG. 10 is an explanatory diagram showing an example of a screen displaying an estimation result of a region of micro content. [Figure 8] FIG. 10 is an explanatory diagram illustrating an overlapping region. [Figure 9] 10 is a flowchart showing a processing procedure for displaying microcontent areas for each area type. [Figure 10]10 is a flowchart showing a processing procedure for extracting a region of microcontent based on a threshold value of estimation accuracy. [Figure 11] 10 is a flowchart showing a processing procedure for extracting a region of microcontent based on a threshold value of the degree of overlap. [Figure 12] 10 is a flowchart showing a processing procedure for accepting changes to micro content. [Figure 13] FIG. 2 is a functional block diagram showing the operation of the computer of the above embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described in detail below with reference to the drawings showing embodiments thereof.

[0010] (Embodiment 1) The first embodiment relates to a form in which artificial intelligence (AI) is used to estimate the area of ​​microcontent in microcontent production. Microcontent is content that is a portion extracted from the entire content, such as an e-book or an article. Note that this embodiment describes an example of microcontent production in a scoring package, but it can also be applied to the production of other types of microcontent (e.g., e-books or articles).

[0011] 1 is an explanatory diagram showing an overview of a content extraction system for micro content production. The system of this embodiment includes an information processing device 1 and an information processing device 2, and each device transmits and receives information via a network N such as the Internet.

[0012] The information processing device 1 is an information processing device that estimates the area of ​​micro content that constitutes content, and displays and transmits the estimated area of ​​micro content. The information processing device 1 is, for example, a server device, a personal computer, or a general-purpose tablet PC (personal computer). In this embodiment, the information processing device 1 is a personal computer that estimates the area of ​​micro content, and will be referred to as a computer 1 below for simplicity.

[0013] The information processing device 2 is an information processing device that performs registration processing of the area of ​​the micro content estimated by the computer 1. The information processing device 2 is, for example, a server device, a personal computer, a general-purpose tablet PC, etc. In this embodiment, the information processing device 2 is assumed to be a server device, and will be referred to as server 2 below for simplicity.

[0014] A computer 1 according to this embodiment acquires a content image including content. When a content image is input, the computer 1 inputs the acquired content image into a learning model that estimates the areas of micro content that make up the content, and estimates the areas of the micro content. The computer 1 displays the estimated areas of each micro content on top of the content.

[0015] 2 is a block diagram showing an example of the configuration of the computer 1. The computer 1 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, and a large-capacity storage unit 17. Each component is connected by a bus B.

[0016] The control unit 11 includes an arithmetic processing unit such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), and GPU (Graphics Processing Unit), and performs various information processing, control processing, and the like related to the computer 1 by reading and executing the control program 1P stored in the storage unit 12. The control program 1P can be deployed so that it is executed on a single computer, or on one site, or distributed across multiple sites and on multiple computers interconnected by a communications network. Although the control unit 11 is described in FIG. 1 as being a single processor, it may also be a multiprocessor.

[0017] The storage unit 12 includes memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory), and stores the control program 1P or data required for the control unit 11 to execute processing. The storage unit 12 also temporarily stores data required for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing processing related to communication.

[0018] The input unit 14 is an input device such as a mouse, keyboard, touch panel, or button, and outputs received operation information to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL (electroluminescence) display, or the like, and displays various information according to instructions from the control unit 11.

[0019] The reading unit 16 reads a portable storage medium 1a including a CD (Compact Disc)-ROM or a DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 16 and store it in the mass storage unit 17. Alternatively, the control unit 11 may download the control program 1P from another computer via a network N or the like and store it in the mass storage unit 17. Furthermore, the control unit 11 may read the control program 1P from the semiconductor memory 1b.

[0020] The mass storage unit 17 includes a recording medium such as an HDD (Hard disk drive) or an SSD (Solid State Drive). The mass storage unit 17 includes an area estimation model 171. The area estimation model 171 is an estimator that estimates the area of ​​micro content that constitutes content based on a content image including the content, and is a trained model generated by machine learning.

[0021] The computer 1 may execute various information processing and control processes on a single computer, or may execute them in a distributed manner on multiple computers, or may execute them in a distributed manner on virtual machines. Note that the various information processing and control processes related to the computer 1 may also be executed on a server device or the like having a communication environment.

[0022] 3 is a block diagram showing an example of the configuration of the server 2. The server 2 includes a control unit 21, a storage unit 22, a communication unit 23, a reading unit 24, and a large-capacity storage unit 25. Each component is connected by a bus B.

[0023] The control unit 21 includes a processing unit such as a CPU, MPU, GPU, etc., and performs various information processing, control processing, etc. related to the server 2 by reading and executing a control program 2P stored in the storage unit 22. The control program 2P can be deployed so that it is executed on a single computer, or on one site, or distributed across multiple sites and on multiple computers interconnected by a communication network. Although the control unit 21 is described in FIG. 3 as being a single processor, it may also be a multiprocessor.

[0024] The storage unit 22 includes memory elements such as RAM and ROM, and stores the control program 2P or data required for the control unit 21 to execute processing. The storage unit 22 also temporarily stores data required for the control unit 21 to execute arithmetic processing. The communication unit 23 is a communication module for performing processing related to communication.

[0025] The reading unit 24 reads the portable storage medium 2a, which includes a CD-ROM or a DVD-ROM. The control unit 21 may read the control program 2P from the portable storage medium 2a via the reading unit 24 and store it in the mass storage unit 25. Alternatively, the control unit 21 may download the control program 2P from another computer via the network N or the like and store it in the mass storage unit 25. Furthermore, the control unit 21 may read the control program 2P from the semiconductor memory 2b.

[0026] The mass storage unit 25 includes a recording medium such as an HDD, an SSD, etc. The mass storage unit 25 includes a micro content DB (database) 251. The micro content DB 251 stores information about micro content that constitutes the content.

[0027] The server 2 may execute various information processes and control processes on a single computer, or may execute the processes in a distributed manner on a plurality of computers, or may execute the processes in a distributed manner on virtual machines.

[0028] 4 is an explanatory diagram showing an example of a record layout of the micro content DB 251. The micro content DB 251 includes a content ID column, a content column, an area ID column, a type column, and a coordinate column. The content ID column stores a unique content ID to identify each content. The content column stores content images including content, or content that is image data such as a PDF (Portable Document Format) file. The area ID column stores area IDs for identifying the areas of the micro content that make up the content.

[0029] The type column stores the type of area of ​​the micro content. The content is data such as an e-book, article, newspaper, or magazine. Micro content is various types of data in text or image units extracted from the entire content, and is content that has been converted into a highly versatile state. The type of area of ​​the micro content is set according to the micro content. For example, if the content is a magazine, the micro content is classified into title, subtitle, figure, and caption. In this embodiment, the content is a scoring package, and the micro content is classified into answer section, question section, and figure. The type of area of ​​the micro content is set to answer section area, question section area, and figure area.

[0030] The coordinate string stores the coordinate values ​​of the microcontent area. For example, the coordinate values ​​of the top left and bottom right of the area are stored in the coordinate string. Note that the coordinate string may also store the coordinate values ​​of the four corners of the area, for example.

[0031] Next, a process for estimating the area of ​​a micro content using the area estimation model 171 will be described. The computer 1 acquires a content image including the content from the mass storage unit 17 or an external device. The computer 1 inputs the acquired content image into the area estimation model 171, which estimates the area of ​​the micro content that constitutes the content, and outputs an estimation result of the area of ​​the micro content.

[0032] FIG. 5 is an explanatory diagram illustrating the region estimation model 171. The region estimation model 171 is used as a program module that is part of artificial intelligence software. The region estimation model 171 is a detector that receives a content image containing content as input and outputs an estimation result of the region of micro content that constitutes the content. In this embodiment, the region estimation model 171 is constructed using RepPoints (Point Set Representation for Object Detection), which is one of the object detection methods.

[0033] RepPoints is an object detection method that does not use anchors, and for an input image, it outputs a rectangular region of an object belonging to a predetermined category. Unlike typical object detection methods such as YOLO (You Only Look Once) or Faster-RCNN (Regions with Convolutional Neural Networks), RepPoints does not use anchors, but instead calculates a set of representative points (RepPoints) that can represent the object region and converts them into a rectangular region.

[0034] Specifically, the computer 1 acquires multiple combinations of training data in which a content image including the content is associated with the type and coordinate values ​​of the area of ​​each micro content constituting the content from the micro content DB 251 of the server 2. The training data is data in which the content image is labeled with the type and coordinate values ​​of the area of ​​each micro content constituting the content. Note that the training data may be collected from a large amount of micro content areas that have been manually extracted in the past.

[0035] The computer 1 performs learning using the acquired training data. Specifically, the computer 1 inputs the content image, which is the training data, into the area estimation model 171 and converts the content image into a feature map. The computer 1 obtains N points representing the area of ​​the micro content from the converted feature map. The computer 1 creates a rectangular area from the obtained N points. The computer 1 calculates the loss (point loss) by evaluation using a loss function (e.g., Focal Loss) defined by the error between the created rectangular area (pseudo rectangular area) and the rectangular area (true rectangular area) of the micro content, which is the training data.

[0036] In this way, the computer 1 performs an estimation area process based on characteristic points of the micro content in the image and a classification process that aggregates the feature amounts obtained from each point, thereby generating an area estimation model 171. The computer 1 can build a possible model for estimating the areas of the micro content that make up the content by learning the area estimation model 171 using training data.

[0037] When the computer 1 acquires a content image including content, the computer 1 inputs the acquired content image to the area estimation model 171. The computer 1 obtains a set of representative points of the micro content area that can express the area of ​​each micro content in the area estimation model 171, and converts the set into a rectangular area of ​​the micro content. The computer 1 acquires the converted micro content area as an output.

[0038] The area estimation model 171 outputs the estimation result of the area of ​​the micro content. The estimation result includes the estimation accuracy, type, and coordinate value of the area of ​​the micro content. The type includes, for example, a question column, an answer column, and a figure. The coordinate value may be, for example, the coordinate value of the top left and the bottom right of the area, or the coordinate values ​​of the four corners of the area. In FIG. 5, the estimation results of the area of ​​the micro content for each of the question column, the answer column, and the figure are output for the content image.

[0039] As shown in the figure, estimated microcontent area 71a has coordinate values ​​of (x1, y1) at the top left and (x1', y1') at the bottom right, the area type is a question column, and the estimation accuracy is 0.92. Also, estimated microcontent area 71b has coordinate values ​​of (x2, y2) at the top left and (x2', y2') at the bottom right, the area type is an answer column, and the estimation accuracy is 0.9. Also, estimated microcontent area 71c has coordinate values ​​of (x3, y3) at the top left and (x3', y3') at the bottom right, the area type is a figure, and the estimation accuracy is 0.95.

[0040] Note that the estimation process of the micro content area is not limited to the above-described RepPoints object detection method. For example, the micro content area may be estimated using an estimation method such as YOLO, Faster-RCNN, or Residual Network (ResNet).

[0041] 6 is a flowchart showing the processing steps for estimating the area of ​​a micro content. The control unit 11 of the computer 1 acquires a content image including the content from the communication unit 13 or the input unit 14 (step S101). The control unit 11 estimates the area of ​​each micro content constituting the content using the area estimation model 171 (step S102). Specifically, the control unit 11 inputs the acquired content image to the area estimation model 171 and outputs the estimation result of the area of ​​the micro content constituting the content.

[0042] The control unit 11 superimposes the estimation result of the micro content area on the content via the display unit 15 (step S103). The estimation result includes the estimation accuracy, type, and coordinate values ​​of each micro content area. The control unit 11 transmits the content ID and the estimation result to the server 2 via the communication unit 13 (step S104). The control unit 21 of the server 2 receives the content ID and the estimation result transmitted from the computer 1 via the communication unit 23 (step S201).

[0043] The control unit 21 stores the received estimation result in the micro content DB 251 of the mass storage unit 25 in association with the content ID (step S202). Specifically, the control unit 21 assigns an area ID to the area of ​​each micro content. The control unit 21 stores the area ID, type, and coordinate values ​​of the assigned micro content in association with the content ID for each micro content area in the micro content DB 251 as one record. The control unit 21 then ends the process.

[0044] According to this embodiment, it is possible to estimate the area of ​​micro content that constitutes a content by using the area estimation model 171 based on a content image that includes the content.

[0045] According to this embodiment, it is possible to estimate content extraction work in micro content production with high accuracy and automatically without relying on human labor.

[0046] According to this embodiment, it is possible to improve the efficiency of content extraction work in micro content production.

[0047] (Embodiment 2) The second embodiment relates to a form in which the estimated area of ​​each micro content is displayed on the content in a different manner depending on the estimation result of the area of ​​the micro content. Note that a description of the content that overlaps with the first embodiment will be omitted.

[0048] The computer 1 acquires the area of ​​each micro content estimated using the area estimation model 171. The computer 1 displays the area of ​​each micro content on the content in a different manner depending on the type of area of ​​each acquired micro content, the estimation accuracy, or the degree of overlap. The degree of overlap will be described later. In addition, the computer 1 can accept corrections or deletions to the estimated area of ​​the micro content, or addition of a new area of ​​the micro content.

[0049] 7 is an explanatory diagram showing an example of a screen displaying the estimation results of the microcontent area. The screen includes a type A area frame 11a, a type B area frame 11b, a type C area frame 11c, a content display field 11d, a type A selection button 12a, a type B selection button 12b, a type C selection button 12c, an add button 13a, a delete button 13b, a modify button 13c, a low score threshold setting field 14a, and a region overlap threshold setting field 14b.

[0050] The type A area frame 11a is an area frame that surrounds the area of ​​micro content whose area type is type A, for each area. The type B area frame 11b is an area frame that surrounds the area of ​​micro content whose area type is type B, for each area. The type C area frame 11c is an area frame that surrounds the area of ​​micro content whose area type is type C, for each area. The content display column 11d is a display column that displays content.

[0051] As shown in the figure, there are three types of microcontent areas: type A, type B, and type C. Type A is a type that indicates the area of ​​the title of the content (microcontent A). Type B is a type that indicates the area of ​​the main text in the content (microcontent B). Type C is a type that indicates the area of ​​a figure in the content (microcontent C). Note that the types of microcontent areas are not limited to those described above, and for example, the types of microcontent areas in the scoring package may be set to an answer column, a question column, and a figure.

[0052] Also, a type A selection button 12a, a type B selection button 12b, and a type C selection button 12c are provided according to the type of micro content area. The type A selection button 12a is a button that accepts the selection of a micro content area whose area type is type A. The type B selection button 12b is a button that accepts the selection of a micro content area whose area type is type B. The type C selection button 12c is a button that accepts the selection of a micro content area whose area type is type C.

[0053] The computer 1 acquires a content image including the content. The computer 1 displays the acquired content image in the content display field 11d. The computer 1 inputs the acquired content image to the area estimation model 171 and acquires an estimation result that estimates the area of ​​each micro content that makes up the content. The computer 1 acquires the type and coordinate values ​​of the area of ​​the micro content included in the estimation result.

[0054] When the computer 1 receives a touch (click) operation on the type A selection button 12a, it extracts the area of ​​type A micro content (content A) from the area of ​​the micro content estimated using the area estimation model 171. Based on the coordinate values ​​of the extracted micro content area, the computer 1 superimposes a type A area frame 11a that surrounds each area of ​​the micro content on the content. In this case, the computer 1 controls to display only the type A area frame 11a and not to display the type B area frame 11b and type C area frame 11c.

[0055] When the computer 1 receives a touch operation of the type B selection button 12b, it extracts the area of ​​the type B micro content from the area of ​​the micro content (content B) estimated using the area estimation model 171. Based on the coordinate values ​​of the extracted micro content area, the computer 1 superimposes and displays a type B area frame 11b that surrounds each area of ​​the micro content on the content. In this case, the computer 1 controls to display only the type B area frame 11b and not to display the type A area frame 11a and type C area frame 11c.

[0056] When the computer 1 receives a touch operation of the type C selection button 12c, it extracts the area of ​​type C micro content (figure) from the area of ​​the micro content estimated using the area estimation model 171. Based on the coordinate values ​​of the extracted micro content area, the computer 1 superimposes a type C area frame 11c that surrounds each area of ​​the micro content on the content. In this case, the computer 1 controls to display only the type C area frame 11c and not to display the type A area frame 11a and type B area frame 11b.

[0057] The display mode (format) of the microcontent area is not limited to a specific format as long as the microcontent area can be recognized by the computer 1. For example, the shape of the border line indicating the microcontent area may be a "solid line," a "dotted line," or a "double line," or the color of the border may be blue, green, red, or the like.

[0058] For example, the computer 1 may set the border of the area of ​​micro content of type A to a dotted line, the border of the area of ​​micro content of type B to a solid line, and the border of the area of ​​micro content of type C to a double line. Alternatively, the computer 1 may set the color of the border of the area of ​​micro content of type A to blue, the color of the border of the area of ​​micro content of type B to yellow, and the color of the border of the area of ​​micro content of type C to red.

[0059] 7 illustrates an example in which a single region type is selected, but the present invention is not limited to this. For example, multiple region types (e.g., type A and type B) may be selected, and microcontent regions corresponding to the multiple selected region types may be simultaneously displayed on the content.

[0060] The add button 13a is a button for adding a new micro content area, the delete button 13b is a button for deleting a micro content area, and the modify button 13c is a button for modifying a micro content area.

[0061] When the computer 1 receives a touch operation on the add button 13a, it generates a frame indicating the area of ​​the microcontent by dragging a pointing device such as a mouse. The computer 1 displays the generated frame on top of the content. In this case, the type of area can be set for the added microcontent area. Specifically, the computer 1 generates a combo box 13d for setting the type of area of ​​the microcontent and displays it on the screen. The combo box 13d is set to allow selection of area types including type A, type B, and type C. The computer 1 receives the setting of the area type through the combo box 13d.

[0062] When the computer 1 receives a touch operation on the delete button 13b, it receives a selection of an area of ​​micro content to be deleted, and deletes the received area of ​​micro content.

[0063] When the computer 1 receives a touch operation of the edit button 13c, it accepts the movement of the position of the area to be edited and the edit of its size or shape (square or rectangle) by dragging and dropping the frame indicating the area of ​​the microcontent. It can also accept edits to the type of area of ​​the microcontent. Specifically, the computer 1 receives the selection of the area to be edited, and with the area selected, generates a combo box 13d that allows the type of area to be selected and displays it on the screen. The computer 1 accepts edits to the type of area to be edited through the combo box 13d.

[0064] The computer 1 transmits the area information (area type, coordinate values, etc.) of the micro content after the change (modification, deletion, or addition) to the server 2. The server 2 receives the area information of the micro content transmitted from the computer 1 and stores it in the micro content DB 251.

[0065] Specifically, the server 2 assigns an area ID to the area of ​​the added micro content. The computer 1 stores the assigned area ID, area type, and coordinate values ​​in association with the content ID as one record in the micro content DB 251. The server 2 deletes the record of the deleted micro content area from the micro content DB 251. The server 2 updates the type and coordinate values ​​of the micro content area for the modified micro content area by associating it with the content ID.

[0066] The low score threshold setting field 14a is a field that accepts the setting of a threshold for the estimation accuracy (score) of a region. As shown in the figure, the low score threshold setting field 14a is a slider bar that accepts the setting of a continuous accuracy value (a value ranging from "0" to "1"). The threshold for the estimation accuracy can be adjusted by moving the slider bar left and right while pressing the left button of the mouse or a finger. Note that, although an example in which the low score threshold setting field 14a is a slider bar has been described in FIG. 7, this is not limiting. For example, the low score threshold setting field 14a may be a text field that accepts input of a threshold for the estimation accuracy.

[0067] When the computer 1 receives a setting operation for the low score threshold setting field 14a, it receives the setting of the threshold for the estimation accuracy. The computer 1 compares the area accuracy of each micro content included in the acquired estimation result with the received threshold for the estimation accuracy. Based on the comparison result, the computer 1 extracts (identifies) the area of ​​the micro content that is equal to or less than the threshold for the estimation accuracy as a low score area. The computer 1 superimposes and displays the extracted low score area on the content.

[0068] Specifically, when the computer 1 receives a selection of an area type, it superimposes low-score areas of the selected area type on the content. For example, if type B is selected, it superimposes a type B area frame 11b surrounding the low-score area of ​​type B on the content. If no type is selected, the computer 1 superimposes low-score areas of all area types (for example, type A, type B, and type C) on the content.

[0069] Furthermore, when the computer 1 receives a change to the set estimation accuracy threshold via the low-score threshold setting field 14a, it updates the low-score area superimposed on the content based on the changed accuracy threshold. Specifically, when the computer 1 receives a change to the estimation accuracy threshold, it re-compares the area accuracy of each microcontent included in the estimation result with the received changed estimation accuracy threshold. Based on the re-comparison result, the computer 1 extracts the area of ​​the microcontent that is equal to or less than the changed estimation accuracy threshold as a low-score area. The computer 1 superimposes the extracted low-score area on the content.

[0070] That is, in accordance with a change in the threshold of the estimation accuracy, the computer 1 updates and redisplays the low-score regions superimposed on the content. The higher the threshold of the estimation accuracy, the more likely it is that low-score regions will be extracted.

[0071] Furthermore, low-score regions can be displayed in different colors. For example, the computer 1 may display low-score regions in red.

[0072] Next, before describing the process of receiving the setting of the threshold value for the degree of overlap, the overlap region will be described. Fig. 8 is an explanatory diagram illustrating the overlap region. Fig. 8A is an explanatory diagram showing an example of a high overlap region. Fig. 8B is an explanatory diagram showing an example of a low overlap region.

[0073] The overlapping area of ​​the microcontent is determined based on the degree of overlap of the microcontent area, which is the ratio of the area of ​​overlap between the microcontent area and the area where other microcontents overlap.

[0074] The overlapping area will be described below based on the degree of overlap of area B with area A. Computer 1 acquires the coordinate values ​​of area A and area B included in the area estimation result. Computer 1 calculates the degree of overlap of area B with area A based on the acquired coordinate values ​​of area A and area B. Computer 1 compares the calculated degree of overlap of area B with a predetermined threshold value for the degree of overlap.

[0075] If the calculated degree of overlap of area B is equal to or greater than a threshold (e.g., 0.6), computer 1 determines it to be a high overlap area. FIG. 8A illustrates an example of a high overlap area. Computer 1 also displays area B in a color that corresponds to the degree of overlap. For example, computer 1 displays the color of the border indicating area B in a color (e.g., red) different from that of area A. If the calculated degree of overlap of area B is less than the threshold, computer 1 determines it to be a low overlap area. FIG. 8B illustrates an example of a low overlap area. Note that in this embodiment, area B determined to be a low overlap area is not subject to color-coded display.

[0076] Next, returning to FIG. 7, the process of accepting the setting of the threshold value for the degree of overlap of regions will be described. The region overlap threshold setting field 14b is a field that accepts the setting of the threshold value for the degree of overlap of regions for determining overlap regions. As shown in the figure, the region overlap threshold setting field 14b accepts the setting of a continuous overlap degree value (a value ranging from "0" to "1"). Note that, although an example in which the region overlap threshold setting field 14b is a slider bar has been described in FIG. 7, this is not limiting. For example, the region overlap threshold setting field 14b may be a text field that accepts input of the threshold value for the degree of overlap.

[0077] When the computer 1 receives a setting operation for the area overlap threshold setting field 14b, it receives the setting of a threshold for the degree of overlap of the microcontent areas. The computer 1 calculates the degree of overlap of each area based on the coordinate values ​​of each microcontent area included in the obtained estimation result. The computer 1 compares the calculated degree of overlap of each area with the received threshold for the degree of overlap.

[0078] The computer 1 extracts (identifies) the area of ​​the microcontent where the degree of overlap is equal to or greater than the threshold value (overlapping area) from the estimation result. The computer 1 superimposes and displays the extracted overlapping area on the content in a color that corresponds to the degree of overlap. For example, the computer 1 displays the frame color of the overlapping area in red.

[0079] Note that multiple overlap degree thresholds can be set. For example, if the calculated overlap degree is equal to or greater than a predetermined first overlap degree threshold (e.g., 0.45) and is less than a predetermined second overlap degree threshold (e.g., 0.75), the computer 1 displays the border line representing the area in yellow. If the calculated overlap degree is equal to or greater than the predetermined second overlap degree threshold, the computer 1 displays the border line representing the area in red.

[0080] Furthermore, when the computer 1 receives a change to the set overlap threshold in the area overlap threshold setting field 14b, it re-extracts the microcontent area whose overlap degree is equal to or greater than the changed overlap threshold. The computer 1 superimposes the re-extracted microcontent area on the content. That is, in response to the change in the overlap threshold, the computer 1 updates and re-displays the superimposed area superimposed on the content. The lower the overlap threshold, the more likely it is that overlap areas with small overlap areas will be extracted.

[0081] It is possible to change the threshold of the estimation accuracy or the threshold of the degree of overlap for each microcontent area. For example, the computer 1 accepts the selection of the microcontent area to be changed. The computer 1 may also accept the setting of the threshold of the estimation accuracy or the threshold of the degree of overlap individually for the selected microcontent area.

[0082] The computer 1 may extract a corresponding area from the estimated multiple microcontent areas based on one or a combination of the type of the microcontent area, the threshold of the estimation accuracy, or the threshold of the degree of overlap. For example, the computer 1 may extract a microcontent area whose type is Type B and whose estimation accuracy is equal to or less than a threshold. The computer 1 displays the extracted microcontent area on top of the content.

[0083] 9 is a flowchart showing the processing steps for displaying micro content regions for each region type. The control unit 11 of the computer 1 acquires a content image including the content (step S111). If the content image is stored in advance in the mass storage unit 17, the control unit 11 acquires the content image from the mass storage unit 17. Note that the control unit 11 may acquire the content image from an external device via the communication unit 13.

[0084] The control unit 11 estimates the area of ​​the micro content based on the acquired content image using the area estimation model 171 (step S112). Specifically, the control unit 11 inputs the acquired content image to the area estimation model 171 and outputs the estimation result of the area of ​​the micro content that constitutes the content.

[0085] The control unit 11 receives a selection of the type of area of ​​the micro content from the input unit 14 (step S113). The control unit 11 extracts a corresponding area from the estimation result according to the received type of area (step S114). For each extracted area, the control unit 11 calculates the degree of overlap between each area and its adjacent area (step S115). The control unit 11 acquires one area from the extracted areas (step S116).

[0086] Control unit 11 determines whether the calculated degree of overlap of each area is equal to or greater than a predetermined threshold value of the degree of overlap (step S117). If control unit 11 determines that the degree of overlap of the area is less than the predetermined threshold value (NO in step S117), control unit 11 displays the area on top of the content via display unit 15 (step S119). Control unit 11 proceeds to the process of step S120, which will be described later.

[0087] When the control unit 11 determines that the degree of overlap of the regions is equal to or greater than a predetermined threshold (YES in step S117), the control unit 11 displays the regions in different colors on the content via the display unit 15 (step S118). For example, the control unit 11 may display the regions in red on the content.

[0088] The control unit 11 determines whether the region is the last region among the extracted regions (step S120). If the control unit 11 determines that the region is not the last region (NO in step S120), the process returns to step S116. If the control unit 11 determines that the region is the last region (YES in step S120), the process ends.

[0089] 10 is a flowchart showing the processing steps for extracting a micro content region based on a threshold value of estimation accuracy. The control unit 11 of the computer 1 acquires a content image including the content (step S131). The control unit 11 estimates the region of the micro content based on the acquired content image using the region estimation model 171 (step S132).

[0090] The control unit 11 receives a setting of a threshold for the estimation accuracy of the micro content area from the input unit 14 (step S133). The control unit 11 extracts the micro content area that is equal to or less than the received threshold for the estimation accuracy from the multiple micro content areas estimated in the process of step S132 (step S134). The control unit 11 displays the extracted micro content area on the display unit 15 (step S135) and ends the process.

[0091] 11 is a flowchart showing the processing steps for extracting a micro content area based on a threshold value of the degree of overlap. The control unit 11 of the computer 1 acquires a content image including the content (step S141). The control unit 11 estimates the area of ​​the micro content based on the acquired content image using the area estimation model 171 (step S142). The control unit 11 receives a setting of a threshold value of the degree of overlap of the micro content area from the input unit 14 (step S143).

[0092] The control unit 11 calculates the degree of overlap of each area based on the coordinate values ​​of each micro content area included in the estimation result (step S144). The control unit 11 extracts, from the estimation result, micro content areas whose calculated overlap degree of the area is equal to or greater than the received overlap degree threshold (step S145). The control unit 11 displays the extracted micro content areas on the display unit 15 (step S146) and ends the process.

[0093] 12 is a flowchart showing the processing procedure when accepting a change to micro content. The control unit 11 of the computer 1 acquires the estimated area of ​​the micro content (step S151). For example, the control unit 11 may estimate the area of ​​the micro content using the area estimation model 171 and acquire the estimated area of ​​the micro content. Alternatively, the control unit 11 may acquire the estimated area of ​​the micro content from the micro content DB 251 of the server 2 via the communication unit 13.

[0094] The control unit 11 displays the acquired micro content areas on the display unit 15 (step S152). The control unit 11 accepts change operations, including correction, deletion, or addition, for each displayed micro content area via the input unit 14 (step S153). The control unit 11 transmits the content ID and area information of the changed micro content (area type, coordinate values, etc.) to the server 2 via the communication unit 13 (step S154).

[0095] The control unit 21 of the server 2 receives the content ID and the changed area information transmitted from the computer 1 via the communication unit 23 (step S251). The control unit 21 stores the received changed area information in the micro content DB 251 of the mass storage unit 25 in association with the received content ID (step S252), and ends the process.

[0096] Furthermore, training data can be created based on the area information of the micro content after the change, and the created training data can be used to retrain the area estimation model 171. Specifically, the computer 1 acquires a content image including the content, and the type and coordinate values ​​of the area after the change of each micro content constituting the content, from the micro content DB 251 of the server 2. The computer 1 creates multiple combinations of training data in which the acquired content image is associated with the type and coordinate values ​​of the area of ​​each micro content constituting the content.

[0097] The computer 1 uses the acquired training data to re-learn the region estimation model 171, similar to the learning process of embodiment 1. Note that the training data is not limited to the changed region information, and may be created based on, for example, a combination of unchanged region information and changed region information.

[0098] In this embodiment, an example has been described in which the area of ​​the micro content is displayed on the computer 1 side, but this is not limiting. For example, the computer 1 transmits the area of ​​the micro content estimated using the area estimation model 171 to the terminal device. The terminal device receives the area of ​​the micro content transmitted from the computer 1. As with the display process described above, the terminal device may display the received area of ​​the micro content on the content in a different manner.

[0099] According to this embodiment, it is possible to display the area of ​​the micro content estimated using the area estimation model 171 on the content in different modes.

[0100] According to this embodiment, it is possible to accept the setting of a threshold value for the estimation accuracy of the microcontent area or a threshold value for the degree of overlap.

[0101] According to this embodiment, it is possible to extract the corresponding microcontent area and display it on top of the content based on the set threshold value of the estimation accuracy of the microcontent area or the threshold value of the degree of overlap.

[0102] According to this embodiment, it is possible to accept changes to the estimated area of ​​the micro content.

[0103] According to this embodiment, the area estimation model 171 is retrained using the changed area of ​​the micro content, thereby making it possible to improve the accuracy of estimating the area of ​​the micro content.

[0104] (Embodiment 3) 13 is a functional block diagram showing the operation of the computer 1 in the above-described embodiment. When the control unit 11 executes the control program 1P, the computer 1 operates as follows.

[0105] The acquisition unit 10a acquires a content image including the content. The estimation unit 10b inputs the content image acquired by the acquisition unit 10a to an area estimation model 171 to estimate the area of ​​each micro content constituting the content. The display unit 10c displays the area of ​​the micro content on the content in different modes depending on the estimation result estimated by the estimation unit 10b.

[0106] The first receiving unit 10d receives modifications or deletions to the microcontent areas estimated by the estimation unit 10b, or the addition of new microcontent areas. The second receiving unit 10e receives settings for a threshold for the estimation accuracy of the microcontent areas. The third receiving unit 10f receives settings for a threshold for the degree of overlap of the microcontent areas. The re-learning unit 10g re-learns the area estimation model 171 based on the content and the changed microcontent areas received by the first receiving unit 10d.

[0107] The first extraction unit 10h extracts microcontent areas where the estimation accuracy of the microcontent areas estimated by the estimation unit 10b is equal to or less than the estimation accuracy threshold accepted by the second acceptance unit 10e. The second extraction unit 10i extracts microcontent areas where the overlapping degree of the microcontent areas estimated by the estimation unit 10b is equal to or greater than the overlapping degree threshold accepted by the third acceptance unit 10f.

[0108] The third embodiment is as described above, and other aspects are the same as those of the first and second embodiments, so the corresponding parts are given the same reference numerals and detailed description thereof will be omitted.

[0109] The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0110] 1. Information processing equipment (computer) 11 Control section 12 Storage section 13 Communications Department 14 Input section 15 Display section 16 Reading unit 17 Mass storage 171 Area Estimation Model 1a Portable storage media 1b semiconductor memory 1P control program 2. Information processing device (server) 21 Control section 22 Memory section 23 Communications Department 24 Reading unit 25 Mass storage 251 Micro Content DB 2a Portable storage media 2b Semiconductor memory 2P control program 10a Acquisition part 10b Estimation part 10c Display section 10d First Reception Section 10e Second Reception Section 10th floor 3rd reception desk 10g Relearning section 10h 1st extraction part 10i 2nd extraction part

Claims

1. an acquisition unit that acquires a content image including the content; an estimation unit that, when a content image is input, inputs the content image acquired by the acquisition unit into a learning model that estimates an area of ​​micro content constituting the content, and estimates an area of ​​the micro content; a second receiving unit that receives a setting for a threshold value of the estimation accuracy of the micro content area; a third receiving unit that receives a setting for a threshold value of the degree of overlap of the microcontent areas; a first extraction unit that extracts a micro content area where the estimation accuracy of the micro content area estimated by the estimation unit is equal to or less than the threshold of the estimation accuracy accepted by the second acceptance unit; a display unit that displays the area of ​​each estimated micro content on the content in a different manner according to the estimation result estimated by the estimation unit; The display unit Displaying the microcontent regions on the content for each type of microcontent region; Displaying the area of ​​the micro content extracted by the first extraction unit on the content; When the degree of overlap of the microcontent regions estimated by the estimation unit is equal to or greater than the threshold value of the degree of overlap received by the third reception unit, the regions are displayed in different colors corresponding to the degree of overlap.

1. An information processing device comprising:

2. a first reception unit that receives a correction or deletion to the area of ​​the micro content estimated by the estimation unit, or an addition of a new area of ​​the micro content; The information processing device according to claim 1 , comprising:

3. a re-learning unit that re-learns the learning model based on the content and the area of ​​the micro content accepted by the first accepting unit; The information processing device according to claim 2 , comprising:

4. The estimation result includes the area of ​​the micro content and the type of the area.

4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

5. a second extraction unit that extracts a microcontent area where the degree of overlap of the microcontent area estimated by the estimation unit is equal to or greater than the threshold value of the degree of overlap accepted by the third acceptance unit; The display unit displays the area of ​​the microcontent extracted by the second extraction unit on the content.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

6. The degree of overlap of the microcontent region is the ratio of the overlapping area between the region and the region overlapping the region.

6. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

7. An information processing system including an information processing device and an information processing terminal, The information processing device includes: an acquisition unit that acquires a content image including the content; an estimation unit that, when a content image is input, inputs the content image acquired by the acquisition unit into a learning model that estimates an area of ​​micro content constituting the content, and estimates an area of ​​the micro content; a second receiving unit that receives a setting for a threshold value of the estimation accuracy of the micro content area; a third receiving unit that receives a setting for a threshold value of the degree of overlap of the microcontent areas; a first extraction unit that extracts a micro content area where the estimation accuracy of the micro content area estimated by the estimation unit is equal to or less than the threshold of the estimation accuracy accepted by the second acceptance unit; a transmission unit that transmits an estimation result estimated by the estimation unit, The information processing terminal a receiving unit that receives the estimation result transmitted by the transmitting unit; a display unit that displays the estimated areas of each micro content on the content in different modes according to the estimation results received by the receiving unit; The display unit Displaying the microcontent regions on the content for each type of microcontent region; Displaying the area of ​​the micro content extracted by the first extraction unit on the content; When the degree of overlap of the microcontent regions estimated by the estimation unit is equal to or greater than the threshold value of the degree of overlap received by the third reception unit, the regions are displayed in different colors corresponding to the degree of overlap. An information processing system comprising:

8. an acquisition unit that acquires a content image including the content; an estimation unit that, when a content image is input, inputs the content image acquired by the acquisition unit into a learning model that estimates an area of ​​micro content constituting the content, and estimates an area of ​​the micro content; a receiving unit that receives a setting for a threshold value of the degree of overlap of the microcontent areas; a display unit that displays the area of ​​each estimated micro content on the content in a different manner according to the estimation result estimated by the estimation unit; When the degree of overlap of the microcontent regions estimated by the estimation unit is equal to or greater than the threshold value of the degree of overlap received by the reception unit, the display unit displays the regions in different colors corresponding to the degree of overlap.

1. An information processing device comprising:

9. An information processing system including an information processing device and an information processing terminal, The information processing device includes: an acquisition unit that acquires a content image including the content; an estimation unit that, when a content image is input, inputs the content image acquired by the acquisition unit into a learning model that estimates an area of ​​micro content constituting the content, and estimates an area of ​​the micro content; a receiving unit that receives a setting for a threshold value of the degree of overlap of the microcontent areas; a transmission unit that transmits an estimation result estimated by the estimation unit, The information processing terminal a receiving unit that receives the estimation result transmitted by the transmitting unit; a display unit that displays the estimated areas of each micro content on the content in different modes according to the estimation results received by the receiving unit; When the degree of overlap of the microcontent regions estimated by the estimation unit is equal to or greater than the threshold value of the degree of overlap received by the reception unit, the display unit displays the regions in different colors corresponding to the degree of overlap. An information processing system comprising:

Images