Information processing device and information processing method

Abstract

Provided is an information processing device that does not reconstruct large-scale information, but instead presents the user with the amount of information the user needs when the user needs it. Provided is an information processing device that outputs information related to a task performed by a responder, namely, task information, wherein the information processing device comprises: an image acquisition unit that acquires an image including a responder and a plurality of responder objects, namely, an original image; an image segmentation unit that segments the original image into a responder image in which the responder is photographed and a plurality of responder object images in which each responder object is photographed; a scene estimation unit that estimates a scene using a first learned model, wherein the first learned model stores an association between an responder image and a scene ID that uniquely represents a scene of an event performed by the responder; and a data block estimation unit that A data block is estimated using one of multiple second learning completion models, the second learning completion model stores the association between multiple images of the object being addressed and one or more data block meta-IDs, the data block meta-ID corresponds one-to-one to the data block ID that uniquely represents the information that divides or prompts the operation information, i.e., the data block; and an output unit, which outputs the data block, the data block estimation unit selects one of the multiple second learning completion models using the model ID that corresponds one-to-one to the scene ID, the data block meta-ID uniquely represents information related to the properties of the object being addressed, i.e., the data block meta-value.

Description

Technical Field

[0001] The present invention relates to an information processing device and an information processing method. Background Art

[0002] For example, in the work assistance system of patent document 1, based on a manual that describes the sequence, content, precautions or other matters of the work, rules that describe the judgment conditions for the object of the work or the status of the work are generated, the object of the work and the status of the work are identified based on sensor information from a device worn by the worker, and work assistance information is output based on the generated rules and the recognition result of the recognition unit.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-109844 Summary of the invention

[0006] Problems to be solved by the invention

[0007] However, in the conventional method described in Patent Document 1, information stored as documents such as manuals can only be retrieved in units of documents. For example, when searching for documents in units of paragraphs, the documents need to be reconstructed into structured information. There are the following problems: considering the cost and effect, it is often unrealistic to reconstruct all the documents that are the search targets. In addition, a large amount of unnecessary information in the information in units of documents will be viewed, and there are cases where the readers of the documents cannot respond quickly.

[0008] An object of one aspect of an embodiment of the present invention is to provide an information processing device that presents a user with information of an amount necessary for the user when the user needs it, without reconfiguring large-scale information.

[0009] Means for solving problems

[0010] Provided is an information processing device that outputs information related to an operation performed by a responder, namely, operation information, wherein the information processing device comprises: an image acquisition unit that acquires an image including a respondent responded to by the responder and a plurality of respondent objects responded to by the responder, namely, an original image; an image segmentation unit that segments the original image into a respondent image in which the respondent is photographed and a plurality of respondent object images in which each respondent object is photographed; a scene estimation unit that estimates a scene of an event performed by the responder using a first learned model, wherein the first learned model stores an association between the respondent image and a scene ID that uniquely represents the scene; and a data block An estimation unit, which estimates a data block as information that divides or prompts the operation information using one of a plurality of second learning completion models, wherein the plurality of second learning completion models store associations between a plurality of images of a corresponding object and one or more data block meta-IDs, and the data block meta-ID corresponds one-to-one to a data block ID that uniquely represents the data block; and an output unit, which outputs the data block, wherein the data block estimation unit selects one of a plurality of second learning completion models using a model ID that corresponds one-to-one to a scene ID, and the data block meta-ID uniquely represents information related to the properties of the corresponding object, i.e., a data block meta-value.

[0011] Provided is an information processing method, which is executed by an information processing device that outputs information related to an operation performed by a responder, namely, operation information, wherein the information processing method has the following steps: a first step of obtaining an image including a responder and a plurality of responder objects, namely, an original image; a second step of segmenting the original image into a responder image in which the responder is photographed and a plurality of responder object images in which respective responder objects are photographed; a third step of estimating a scene using a first learning completion model, wherein the first learning completion model stores the association between the responder image and a scene ID that uniquely represents the scene; a fourth step of estimating a data block using one of a plurality of second learning completion models, wherein the plurality of second learning completion models stores the association between a plurality of responder object images and one or more data block meta-IDs, wherein the data block meta-IDs correspond one-to-one to the data block IDs that uniquely represent the data blocks; and a fifth step of outputting a data block, wherein one of a plurality of second learning completion models is selected using a model ID that corresponds one-to-one to the scene ID, wherein the data block meta-ID uniquely represents information related to the properties of the responder object, namely, a data block meta-value.

[0012] Effects of the Invention

[0013] According to one aspect of the embodiment of the present invention, it is possible to realize an information processing device that presents a user with information of an amount necessary for the user when the user needs it, without reconfiguring large-scale information. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a block diagram showing the configuration of an information processing device in the utilization stage of the present embodiment.

[0015] Figure 2 : is a block diagram showing the configuration of the information processing device in the learning phase of the present embodiment.

[0016] Figure 3 1 is a diagram showing an original image, a treated person image, and a plurality of treated object images according to the present embodiment.

[0017] Figure 4 This is a diagram showing the relationship between the response object image and a plurality of response object images according to the present embodiment, that is, a tree structure.

[0018] Figure 5 1 is a diagram showing a first learned model and a second learned model according to the present embodiment.

[0019] Figure 6 It is a diagram showing information stored in the auxiliary storage device according to the present embodiment.

[0020] Figure 7 It is a sequence diagram for explaining the scene estimation function, the data block estimation function, and the data block output function of the present embodiment.

[0021] Figure 8 This is a sequence diagram for explaining the first learned model generation function and the second learned model generation function of the present embodiment.

[0022] Fig. 9 This is a flowchart showing the processing procedure of information processing in the utilization phase of the present embodiment.

[0023] Fig.10 : is a flowchart showing the processing procedure of information processing in the learning phase of this embodiment. DETAILED DESCRIPTION

[0024] Hereinafter, one embodiment of the present invention will be described in detail using the accompanying drawings. For example, information on the object to be handled that is referred to by an operator, i.e., a responder, who handles a respondent such as a student, a student's protector, or a patient at a window at a university or a window at a pharmacy is described. For example, in the case of a window at a university, the object to be handled is a document, and in the case of a window at a pharmacy, the object to be handled is a medicine.

[0025] (Present Embodiment)

[0026] First, use Figure 1 The information processing device 1 in the utilization phase will be described. Figure 11 is a block diagram showing the configuration of the information processing device 1 in the utilization stage of the present embodiment. The information processing device 1 includes a central processing unit 2 , a main storage device 3 , and an auxiliary storage device 11 .

[0027] The central processing unit 2 is, for example, a CPU (Central Processing Unit), and executes processing by calling a program stored in the main storage device 3. The main storage device 3 is, for example, a RAM (Random Access Memory), and stores programs such as an image acquisition unit 4, an image segmentation unit 5, a scene estimation unit 6, a data block estimation unit 7, a data block output unit 8, a first learning completed model generation unit 9, a second learning completed model generation unit 10, and a recommended image output unit 13 described later.

[0028] In addition, the program including the image acquisition unit 4, the image segmentation unit 5, the scene estimation unit 6, the data block estimation unit 7, the data block output unit 8 and the selected image output unit 13 can also be referred to as the control unit 15, and the program including the first learning completion model generation unit 9 and the second learning completion model generation unit 10 can also be referred to as the learning completion model generation unit 16.

[0029] The auxiliary storage device 11 is, for example, an SSD (Solid State Drive) or an HDD (Hard Disk Drive), and stores databases such as the first learning completed model DB1, the first learning model DB1', the second learning completed model DB2, and the second learning model DB2' described later; and tables such as the scene table TB1, the model table TB2, the content table TB3, the scene / content table TB4, the content / data block table TB5, the data block / meta table TB6, the data block table TB7, and the data block meta table TB8.

[0030] like Figure 1 As shown, the information processing device 1 that outputs information related to the work performed by the respondent, that is, work information, comprises, in the use stage, an image acquisition unit 4, an image segmentation unit 5, a scene estimation unit 6, a data block estimation unit 7, and a data block output unit 8 that outputs data blocks that are information obtained by segmenting or presenting the work information. Here, the work information may also be referred to as content, and the content ID uniquely indicates the work information.

[0031] The image acquisition unit 4 acquires the image of the respondent 21 ( Figure 3 ) and the multiple objects 22 to 25 ( Figure 3 ) is the original image 20( Figure 3The image segmentation unit 5 segments the original image 20 into a subject image 30 in which the subject 21 is photographed and a plurality of subject object images 40 to 43 in which respective subjects 22 to 25 are photographed.

[0032] The scene estimation unit 6 estimates the situation implemented by the responder, that is, the scene. Specifically, the scene estimation unit 6 obtains the responder image 30 (35) and estimates the scene using the first learned model DB1, which stores the association between the responder image 30 (35) and the scene ID that uniquely represents the scene.

[0033] The scene estimation unit 6 uses the scene ID as a search key, obtains the scene name from the scene table TB1 in which the scene ID and the scene name are associated one-to-one, and sends it to the user terminal 12. The user terminal 12 presents the scene name received from the scene estimation unit 6 to the user.

[0034] The data block estimation unit 7 obtains an image of the object 22 (23~25) related to the operation, namely, the object images 40~43, and estimates the data block using one of the multiple second learning completion model DB2s. The second learning completion model DB2 stores the association between the object image 40 (41~43) and one or more data block meta-IDs, and the data block meta-ID corresponds one-to-one to the data block ID that uniquely represents the data block.

[0035] The data block estimation unit 7 selects one second learned model DB 2 among the plurality of second learned model DB 2 using the model ID corresponding to the scene ID in a one-to-one manner. The data block meta ID uniquely indicates a data block meta value that is information on the properties of the objects 22 to 25.

[0036] The data block estimation unit 7 uses the scene ID as a search key to obtain the model ID from the model table TB2, a table in which the model ID and the scene ID are associated one-to-one. In addition, the data block estimation unit 7 uses the data block meta ID as a search key to obtain the data block ID from the data block / meta table TB6, a table in which the data block ID and the data block meta ID are associated one-to-one or one-to-many.

[0037] Furthermore, the block estimation unit 7 uses the block ID as a search key to obtain a block summary indicating an outline of the block from the block table TB7, and transmits the block summary to the user terminal 12. The user terminal 12 presents the block summary received from the block estimation unit 7 to the user.

[0038] Furthermore, the data block estimation unit 7 uses the data block ID as a search key, obtains the data block from the data block table TB7, and transmits the data block to the user terminal 12. The user terminal 12 presents the data block received from the data block estimation unit 7 to the user.

[0039] The data block table TB7 is a table in which data blocks, data block digests, and hash values ​​are associated with data block IDs one-to-one. The hash value is used, for example, to confirm whether a data block has been changed.

[0040] The information processing device 1 may further include a recommended image output unit 13, and the auxiliary storage device 11 may further include a recommended table TB9. The recommended image output unit 13 uses a combination of a model ID and one or more data block meta IDs as a search key and uses the recommended object image using the recommended table TB9.

[0041] The recommended image output unit 13 outputs the retrieved recommended corresponding object image to the user terminal 12. The recommended corresponding object image is an image that is not captured in the original image 20 but can be inferred as the originally required corresponding object. In addition, the recommended table TB9 is a table that associates a combination of a model ID and a data block meta ID with a recommended corresponding object image one-to-one.

[0042] Next, use Figure 2 The information processing device 1 in the learning phase is described. For example, in the learning phase, a person image 30 (35) and one or more object images 40 to 43 input from an input device (not shown) are learned as a set. Here, learning refers to learning with a teacher, for example.

[0043] Figure 2 1 is a block diagram showing the configuration of the information processing device 1 in the learning phase of the present embodiment. In the learning phase, the information processing device 1 includes a first learned model generation unit 9 and a second learned model generation unit 10 .

[0044] The first learned model generating unit 9 is a program that generates the first learned model DB1 by learning the first learned model DB1 ′ using the scene ID and the person being treated image 30 ( 35 ) as a pair.

[0045] The first learned model generating unit 9 obtains the scene ID from the scene table TB1 for the treated person image 30 ( 35 ), and obtains the model ID corresponding to the scene ID from the model table TB2 .

[0046] The second learned model generator 10 is a program that specifies a model ID, pairs one or more data block meta-IDs and the corresponding object images 40 (41 to 43), and learns the second learned model DB2' to generate the second learned model DB2.

[0047] The second learning completed model generation unit 10 uses the scene ID as a search keyword and obtains the content ID from the scene / content table TB4, a table in which the scene ID and the content ID are associated in a one-to-many manner. In addition, here, the scene ID as the search keyword is associated with the respondent image 30 (35) paired with the respondent object image 40 (41 to 43) as the processing target.

[0048] The second learned model generation unit 10 uses the content ID as a search key to obtain content from the content table TB3 , which is a table in which content IDs and content are associated one-to-one.

[0049] The second learned model generation unit 10 uses the content ID as a search key to obtain the data block ID from the content / data block table TB5 , which is a table in which the content ID and the data block ID are associated in a one-to-one or one-to-many manner.

[0050] The second learned model generating unit 10 obtains a data block from the data block table TB7 using the data block ID as a search key, and obtains a meta ID for the data block from the data block / meta table TB6 using the data block ID as a search key.

[0051] The second learned model generator 10 uses the block meta ID as a search key to obtain the block meta value from the block meta table TB8. The block meta table TB8 associates the block type ID, block type name, and block meta value with the block meta ID in a one-to-one relationship.

[0052] The data block category ID uniquely indicates the name of the category to which the data block element value belongs, that is, the data block category name. In addition, the second learned model generation unit 10 refers to the object image 40 (41 to 43) to confirm that there is no problem with the acquired data block, content, and data block element value.

[0053] By determining the problematic value as an abnormal value and not using it for the learning with the teacher, the second learned model generating unit 10 can generate a highly accurate learned model DB2, and the information processing device 1 can perform highly accurate processing in the utilization stage.

[0054] Next, use Figure 3The original image 20 acquired by the user terminal 12 and processed as information by the information processing device 1 , and the treated person image 30 and treated object images 40 to 43 generated by segmenting the original image 20 will be described. Figure 3 1 and 2 are diagrams showing an original image 20, a treated person image 30, and a plurality of treated object images 40 to 43 according to the present embodiment.

[0055] The original image 20, the image of the person to be treated 30, and the plurality of images of the object to be treated 40 to 43 are displayed on the user terminal 12, for example. Figure 3 , an example of simultaneous display is shown in FIG. 1 , but the original image 20 , the image of the person to be treated 30 , and the plurality of images of the object to be treated 40 to 43 may be displayed separately on the user terminal 12 .

[0056] The original image 20 captures a person 21 and objects 22 to 25. The sizes of the objects 22 to 25 are estimated based on information that does not change depending on the scene in the cubicle, such as a table. In addition, for the objects 22 to 25, information such as an attached photo 26, internal text 27, and a signature 28 can also be obtained, as with the object 24.

[0057] Next, use Figure 4 The relationship between the respondent 21 and the plurality of respondent objects 22 to 25 will be described. Figure 4 1 is a diagram showing a tree structure of the relationship between the respondent 21 and a plurality of respondent objects 22 to 25 according to the present embodiment.

[0058] like Figure 4 As shown, the image segmentation unit 5 associates the respondent 21 and the plurality of respondent objects 22 to 25 in a tree structure in which the respondent 21 is a root node and the plurality of respondent objects 22 to 25 are leaf nodes or internal nodes.

[0059] Furthermore, the image segmentation unit 5 may further acquire information such as an attached photo 26 , a text 27 , and a signature 28 as information included in at least one of the objects 22 to 25 , and associate them with the tree structure as leaf nodes.

[0060] Next, use Figure 5 The first learned model DB1 and the second learned model DB2 are described. Figure 5 The first learned model DB1 and the second learned model DB2 according to the present embodiment are shown.

[0061] The first learned model DB1 stores associations between a plurality of patient images 30 (35) and a plurality of scene IDs generated by machine learning using a plurality of pairs of patient images 30 (35) and scene IDs as a pair of learning data. Here, the machine learning is, for example, a convolutional neural network (CNN).

[0062] Specifically, the association between the respondent image 30 (35) and the scene ID is represented by the following convolutional neural network, which is composed of Figure 5 The nodes shown in circles, the edges shown in arrows, and the weight coefficients set on the edges are represented. Figure 5 As shown, the input of the patient image 30 ( 35 ) to the first learned model DB 1 is set to each pixel such as pixels p1 and p2 , for example.

[0063] The second learning completed model DB2 is associated with the model ID one-to-one and is set to multiple. Each second learning completed model DB2 stores the association between multiple object images 40 (41-43) and multiple, one or more data block meta-IDs generated by machine learning using multiple pairs of object images 40 (41-43) and one or more data block meta-IDs as a pair of learning data. Here, machine learning is set to, for example, a convolutional neural network (CNN: Convolution Neural Network).

[0064] Specifically, the association between a plurality of object images 40 (41 to 43) and a plurality of, one or more data block meta-IDs can be represented by the following convolutional neural network, which is composed of Figure 5 The nodes shown in circles, the edges shown in arrows, and the weight coefficients set on the edges are represented. Figure 5 As shown, the input of the object image 40 ( 41 to 43 ) to the second learned model DB2 is set to be each pixel such as pixels p1 and p2 , for example.

[0065] Next, use Figure 6 The scene table TB1, model table TB2, content table TB3, scene / content table TB4, content / data block table TB5, data block / meta table TB6, data block table TB7, data block meta table TB8 and recommendation table TB9, which are information stored in the auxiliary storage device 11, are described. Figure 6 It is a diagram showing information stored in the auxiliary storage device 11 according to the present embodiment.

[0066] The scenario ID stored in the scenario table TB1, etc. is set to a 3-digit hexadecimal number such as OFD, etc. In addition, the scenario name stored in the scenario table TB1, etc. is set to, for example, inquiry about grades, consultation on development direction, etc.

[0067] The model ID stored in the model table TB2 is represented by a two-letter alphabet and a one-digit decimal number, for example, as in MD1. The content ID stored in the content table TB3 is represented by a five-digit hexadecimal number and a two-digit decimal number, for example, as in 1B827-01. The content stored in the content table TB3 is represented by a content ID, i.e., a file name, with an extension, such as 1B827-01.txt, and a pointer to the content entity is stored.

[0068] The block ID stored in the content / block table TB5 is represented by 5-digit and 2-digit decimal numbers such as 82700-01. The block meta ID stored in the block / meta table TB6 is represented by 4-digit hexadecimal numbers such as 24FD.

[0069] The data blocks stored in the data block table TB7 are represented by the file name of the content corresponding to the target data block and a 1-digit decimal number, such as 1B827-01.txt_0, and store a pointer to a part of the content entity corresponding to the target data block.

[0070] The data block summary stored in the data block table TB7 is, for example, a document summarizing the content of a data block such as "At an employment agency..." The hash value stored in the data block table TB7 is, for example, a 15-digit hexadecimal number such as 564544d8f0b746e.

[0071] The data block type ID stored in the data block meta table TB8 is set to a 3-digit decimal number such as 394. The data block type name stored in the data block meta table TB8 is set to, for example, the size of the paper, the color of the paper, whether there are holes in the paper, etc. The data block meta values ​​stored in the data block meta table TB8 are set to, for example, A4, B4, white, blue, horizontal holes, no holes, etc. In addition, the values ​​of the data block type ID and the data block type name may be NULL.

[0072] The combination of the data block meta ID stored in the recommendation table TB9 is set to (24FD, 83D9), (25FD), etc., which is a combination of one or more data block meta IDs. The recommended object image stored in the recommendation table TB9 stores, for example, a pointer to an entity showing a file name with an extension such as IMG001.jpg.

[0073] As shown in the scene / content table TB4, the content / data block table TB5 and the data block / meta table TB6, the data structure of the job information has a hierarchical structure in which the data block meta ID is set as the bottom layer, i.e., the first layer, the data block ID is set as the second layer, the content ID is set as the third layer, and the scene ID is set as the top layer, i.e., the fourth layer.

[0074] Next, use Figure 7 The scene estimation function, the data block estimation function, and the data block output function are described. Figure 7 It is a sequence diagram for explaining the scene estimation function, the data block estimation function, and the data block output function of the present embodiment.

[0075] The information processing function in the utilization phase is constituted by a scene estimation function implemented by a scene estimation process S60 described later, a block estimation function implemented by a block estimation process S80 described later, and a block output function implemented by a block output process S100 described later.

[0076] First, the scene estimation function is described. The image acquisition unit 4 included in the control unit 15 receives the original image 20 from the user terminal 12 (S1). Then, the image segmentation unit 5 included in the control unit 15 segments the original image 20 into the person image 30 and the object images 40 to 43.

[0077] The image segmentation unit 5 sends the respondent image 30 to the scene estimation unit 6 and sends the respondent object images 40 to 43 to the data block estimation unit 7. Next, the scene estimation unit 6 included in the control unit 15 inputs the respondent image 30 into the first learned model DB1 (S2).

[0078] The first learned model DB 1 selects one or more scene IDs closely related to the received patient image 30 , and outputs the selected one or more scene IDs (hereinafter, may also be referred to as a first scene ID list) to the scene estimation unit 6 ( S3 ).

[0079] When the scene estimation unit 6 obtains the first scene ID list, it directly sends it to the user terminal 12 (S4). The user terminal 12 sends the scene estimation unit 6 whether there is a cache for each scene ID included in the first scene ID list (S5).

[0080] The user terminal 12 stores the same table as the scene table TB1 for information processed in the past. The user terminal 12 uses the scene ID of the received first scene ID list as a search keyword to search in the table stored in the user terminal 12. The scene IDs that are searched are cached, and the scene IDs that are not searched are not cached.

[0081] The scene estimation unit 6 uses one or more scene IDs (hereinafter also referred to as the second scene ID list) among the scene IDs included in the first scene ID list received from the user terminal 12 and not cached in the user terminal 12 as search keywords to search in the scene table TB1 (S6).

[0082] As a result of the search, the scene estimation unit 6 obtains, from the scene table TB1 , scene names corresponding to the respective scene IDs included in the second scene ID list (hereinafter, this may also be referred to as a scene name list) ( S7 ).

[0083] The scene estimation unit 6 directly transmits the acquired scene name list to the user terminal 12 (S8). In the utilization phase, the information processing device 1 implements a scene estimation function of estimating the scene of the respondent image 30 by estimating the scene name through steps S1 to S8.

[0084] Next, the data block estimation function is described. The user terminal 12 presents the received scene name list to the user. The user selects, for example, one scene name from the presented scene name list. The user terminal 12 sends the scene name selected by the user to the data block estimation unit 7 included in the control unit 15 (S9).

[0085] The data block estimation unit 7 uses the scene ID corresponding to the scene name received from the user terminal 12 as a search key ( S10 ), searches the model table TB2 , and acquires the model ID ( S11 ).

[0086] The data block estimation unit 7 receives the object image 40 (41 to 43) from the image segmentation unit 5 (S12). The data block estimation unit 7 specifies one of the plurality of second learned model DB2s using the model ID obtained from the model table TB2, and inputs the object image 40 (41 to 43) to the specified second learned model DB2 (S13).

[0087] The second learning completion model DB2 selects one or more data block meta-IDs from one or more data block meta-IDs that are closely related to the object image 40 (41 to 43), and outputs the selected one or more, one or more data block meta-IDs (hereinafter, also referred to as a data block meta-ID list) to the data block estimation unit 7 (S14).

[0088] The data block estimation unit 7 searches the data block / meta table TB6 using one or more data block meta IDs included in the data block meta ID list as search keys (S15).

[0089] The data block estimation unit 7 obtains one or more data block IDs (hereinafter, referred to as a first data block ID list) from the data block / meta table TB6 as a search result (S16). The data block estimation unit 7 directly transmits the obtained first data block ID list to the user terminal 12 (S17).

[0090] The user terminal 12 sends the information whether there is a buffer associated with each block ID included in the first block ID list to the block estimation unit 7 (S18). The user terminal 12 stores a table having a block ID column and a block summary column in the block table TB7 for information processed in the past.

[0091] The user terminal 12 uses the block ID of the received first block ID list as a search key to search the table stored in the user terminal 12. The searched block IDs are cached, while the unsearched block IDs are not cached.

[0092] The data block estimation unit 7 uses one or more data block IDs (hereinafter also referred to as the second data block ID list) from among the data block IDs included in the first data block ID list received from the user terminal 12 and not cached in the user terminal 12 as search keywords, and searches in the data block table TB7 (S19).

[0093] The data block estimation unit 7 obtains the data block digests corresponding to the data block IDs included in the second data block ID list (hereinafter, may also be referred to as the data block digest list) from the data block table TB7 as the search result (S20). The data block estimation unit 7 directly transmits the obtained data block digest list to the user terminal 12 (S21).

[0094] In the utilization phase, the information processing device 1 implements a block estimation function of estimating the data blocks of the object 22 (23 to 25) by estimating the block digests through steps S9 to S21.

[0095] Next, the data block output function is described. The user terminal 12 presents the received data block summary list to the user. The user selects, for example, one data block summary from the presented data block summary list. The user terminal 12 sends the data block summary selected by the user to the data block output unit 8 included in the control unit 15 (S22).

[0096] The data block output unit 8 uses the data block ID corresponding to the data block digest received from the user terminal 12 as a search key ( S23 ), searches the data block table TB7 , and acquires the data block ( S24 ).

[0097] The data block output unit 8 directly sends the acquired data block to the user terminal 12 (S25). The user terminal 12 presents the received data block to the user. In the utilization phase, the information processing device 1 implements the data block output function of outputting the data block of the object 22 (23-25) through steps S22-S25.

[0098] Next, use Figure 8 The first learned model generation function and the second learned model generation function are described. Figure 7 This is a sequence diagram for explaining the first learned model generation function and the second learned model generation function of the present embodiment.

[0099] The information processing function in the learning phase is composed of a first learned model generation function implemented by the first learned model generation process and a second learned model generation function implemented by the second learned model generation process.

[0100] First, the first learning completion model generation function is described. The first learning completion model generation unit 9 included in the learning completion model generation unit 16 determines the scene name, the group of the person image 30 and one or more object images 40 to 43 to be processed, and uses the scene name as a search keyword to search the pre-generated scene table TB1 in the scene table TB1 (S31).

[0101] The first learned model generating unit 9 acquires the scene ID from the scene table TB1 as a search result ( S32 ), and causes the first learned model DB1 ′ to perform learning using the respondent image 30 and the scene ID as a pair ( S33 ).

[0102] Furthermore, the first learned model generating unit 9 sends the acquired scenario ID to the model table TB2 to request acquisition of the model ID (S34). The model table TB2 generates a model ID corresponding to the received scenario ID and stores a combination of the scenario ID and the model ID.

[0103] Next, the first learned model generating unit 9 obtains the model ID from the model table TB2 ( S35 ). In the learning phase, the information processing device 1 implements a first learned model generating function of generating the first learned model DB1 through steps S31 to S35 .

[0104] Next, the second learned model generation function will be described. The second learned model generation unit 10 included in the learned model generation unit 16 uses the scenario ID received by the first learned model generation unit 9 in step S32 as a search key to search the pre-generated scenario / content table TB4 (S36).

[0105] The second learned model generation unit 10 acquires the content ID from the scenario / content table TB4 as a search result ( S37 ), and searches the pre-generated content table TB3 using the acquired content ID as a search key ( S38 ).

[0106] The second learned model generating unit 10 acquires content from the content table TB3 as a search result ( S39 ), and searches the pre-generated content / data block table TB5 using the content ID acquired in step S37 as a search key ( S40 ).

[0107] The second learned model generating unit 10 acquires a data block ID as a search result from the content / data block table TB5 ( S41 ), and searches the pre-generated data block table TB7 using the acquired data block ID as a search key ( S42 ).

[0108] The second learned model generating unit 10 acquires a data block from the data block table TB7 as a search result ( S43 ), and searches the pre-generated data block / meta table TB6 using the data block ID acquired in step S41 as a search key ( S44 ).

[0109] The second learned model generator 10 obtains one or more data block meta IDs from the data block / meta table TB6 as search results ( S45 ), and searches the pre-generated data block meta table TB8 using each of the obtained data block meta IDs as a search key ( S46 ).

[0110] The second learned model generating unit 10 obtains data block meta values ​​corresponding to the respective data block meta IDs from the data block meta table TB8 as search results ( S47 ).

[0111] The second learned model generator 10 refers to the respondent image 30 and the respondent object images 40 to 43 to check whether there is a problem with the content acquired in step S39 , the data block acquired in step S43 , and the element value for each data block acquired in step S47 .

[0112] For example, the second learning completed model generation unit 10 checks the expression of the respondent 21 and the document names recorded in the respondent objects 22 to 25. For example, the second learning completed model generation unit 10 determines the expression of the respondent 21 based on the respondent image 30 and determines the document names recorded in the respondent objects 22 to 25 based on the respondent object images 40 to 43.

[0113] If the result of the reference clearly indicates that there is a problem, the processing related to the target group is terminated. The problem is that the content, data block, and data block meta value are information related to a document that is significantly different from the document photographed in the object images 40 to 43.

[0114] Next, the second learned model generation unit 10 uses the model ID, the object image 40 (41-43) and one or more data blocks as a pair, and learns the second learned model DB2' (S48). In the learning stage, the information processing device 1 implements the second learned model generation function of generating the second learned model DB2 through steps S36-S48.

[0115] Next, use Fig. 9 The information processing in the utilization phase is described. Fig. 9 1 is a flowchart showing the processing procedure of the information processing in the utilization phase of the present embodiment. The information processing in the utilization phase is composed of a scene estimation process S60, a data block estimation process S80, and a data block output process S100.

[0116] First, the scene estimation process S60 is described. The scene estimation process S60 is composed of steps S61 to S67. When the scene estimation unit 6 receives the patient image 30 (35) from the image segmentation unit 5 (S61), it inputs the patient image 30 (35) into the first learned model DB1 (S62).

[0117] The scene estimation unit 6 obtains the first scene ID list from the first learned model DB 1 as an output ( S63 ), directly sends the first scene ID list to the user terminal 12 , and inquires whether the user terminal 12 has a cache ( S64 ).

[0118] When all the answer results from the user terminal 12 are in the cache (S65: No), the scene estimation process S60 ends and the data block estimation process S80 starts. When none of the answer results from the user terminal 12 are included in the cache (S65: Yes), the scene estimation unit 6 obtains the scene name list from the scene table TB1 (S66), directly sends it to the user terminal 12 (S67), and the scene estimation process S60 ends.

[0119] Next, the block estimation process S80 will be described. The block estimation process S80 is composed of steps S81 to S88. The block estimation unit 7 receives the scene name selected by the user from the user terminal 12 (S81).

[0120] When receiving the scene name from the user terminal 12, the data block estimation unit 7 obtains the model ID from the model table TB2 (S82). Next, the data block estimation unit 7 specifies one of the plurality of second learned model DB2s by the model ID, and inputs the object image 40 (41 to 43) received from the image segmentation unit 5 into the specified second learned model DB2 (S83).

[0121] The data block estimation unit 7 obtains the data block meta ID list from the second learned model DB2 as an output (S84), and obtains the first data block ID list from the data block / meta table TB6 (S85). Next, the data block estimation unit 7 directly sends the first data block ID list to the user terminal 12, and inquires whether the user terminal 12 has a cache (S86).

[0122] When all the answer results from the user terminal 12 are in the buffer (S86: No), the data block estimation process S80 ends and the data block output process S100 starts. When all the answer results from the user terminal 12 are contained in the buffer (S86: Yes), the data block estimation unit 7 obtains the data block summary list from the data block table TB7 (S87), directly sends it to the user terminal 12 (S88), and the data block estimation process S80 ends.

[0123] Next, the data block output process S100 will be described. The data block output process S100 is composed of steps S101 to S103. The data block output unit 8 receives a data block digest selected by a user from the user terminal 12 (S101).

[0124] When receiving the data block digest from the user terminal 12, the data block output unit 8 obtains the data block from the data block table TB7 (S102), and directly transmits it to the user terminal 12 (S103), and the data block output process S100 ends.

[0125] Next, use Fig.10 The information processing in the learning phase is explained. Fig.10 1 is a flowchart showing the processing procedure of information processing in the learning phase of the present embodiment. The information processing in the learning phase is composed of a first learned model generation process S120 and a second learned model generation process S140.

[0126] First, the first learning completed model generation process S120 is described. The first learning completed model generation process S120 is composed of steps S121 to S124. When determining a group of a scene name, a person image 30 (35) and one or more object images 40 (41 to 43), the first learning completed model generation unit 9 uses the scene name as a search keyword and searches the scene table TB1 (S121).

[0127] The first learned model generating unit 9 obtains the scene ID from the scene table TB1 as a search result ( S122 ), and performs learning on the first learned model DB1 ′ using the scene ID and the respondent image 30 ( 35 ) as a pair ( S123 ).

[0128] Next, the first learned model generating unit 9 sends the scenario ID acquired in step S122 to the model table TB2 to request acquisition of the model ID, and acquires the model ID ( S124 ).

[0129] Next, the second learned model generation process S140 is described. The second learned model generation process S140 is composed of steps S141 to S150. The second learned model generation unit 10 searches the scene / content table TB4 using the scene ID acquired in step S122 as a search key, and acquires a content ID (S141).

[0130] The second learned model generation unit 10 searches the content table TB3 using the obtained content ID as a search key to obtain the content (S142). In addition, the second learned model generation unit 10 searches the content / data block table TB5 using the obtained content ID as a search key to obtain the data block ID (S143).

[0131] Furthermore, the second learned model generation unit 10 uses the acquired data block ID as a search key to search the data block table TB7 and acquire the data block (S144). Furthermore, the second learned model generation unit 10 uses the acquired data block ID as a search key to search the data block / meta table TB6 and acquire one or more meta IDs for the data block (S145).

[0132] Furthermore, the second learned model generating unit 10 uses the acquired one or more data block meta IDs as search keys to search the data block meta table TB8 and acquires data block meta values ​​corresponding to the respective data block meta IDs ( S146 ).

[0133] The second learning completed model generating unit 10 refers to the respondent image 30 (35) and the respondent object image 40 (41-43) to confirm whether there is any problem with the content obtained in step S142, the data block obtained in step S144 and the element value of each data block obtained in step S146 (S147).

[0134] If the result of the confirmation is that there is a problem (S148: No), the information processing of the learning stage related to the group being processed is terminated. If the result of the confirmation is that there is no problem (S148: Yes), the second learning model generation unit 10 uses the model ID, one or more data block element IDs and the corresponding object image 40 (41 to 43) as a pair, and causes the second learning model DB2' to learn (S149), and the information processing of the learning stage related to the group being processed is terminated.

[0135] As described above, according to the information processing device 1 of this embodiment, the data blocks for dividing or prompting the job information are displayed via the user terminal 12. Therefore, by appropriately setting the data blocks, the necessary amount of information can be prompted. In addition, if the data blocks are set to prompt such information for the entire document, there is no need to reconstruct large-scale information.

[0136] By using the model table TB2 , when the relationship between the first learned model DB1 and the second learned model DB2 changes, it is possible to cope with the change by simply changing the model table TB2 , thereby providing a device with excellent maintainability.

[0137] Furthermore, when the model table TB2 is not used, if the relationship between the first learned model DB1 and the second learned model DB2 changes, it is necessary to generate the learned model DB2 again.

[0138] In the present embodiment, the image acquisition unit 4, the image segmentation unit 5, the scene estimation unit 6, the data block estimation unit 7, the data block output unit 8, the first learning completion model generation unit 9, the second learning completion model generation unit 10 and the recommended image output unit 13 are set as programs, but are not limited to this, and can also be logic circuits.

[0139] In addition, the image acquisition unit 4, the image segmentation unit 5, the scene estimation unit 6, the data block estimation unit 7, the data block output unit 8, the first learned model generation unit 9 and the second learned model generation unit 10, the recommended image output unit 13, the first learned model DB1, the first learned model DB1', the second learned model DB2, the second learned model DB2', the scene table TB1, the model table TB2, the content table TB3, the scene / content table TB4, the content / data block table TB5, the data block / meta table TB6, the data block table TB7, the data block meta table TB8 and the recommended table TB9 may not be installed in one device, but may be installed in a distributed manner in multiple devices connected via a network.

[0140] In addition, in the above Figure 8 and Fig.10 The illustrated learning phase describes a case where the first learned model and the second learned model are generated in association with each other, but the present invention is not limited thereto, and the first learned model DB1 and the second learned model DB2 may be generated separately.

[0141] When the first learned model DB1 and the second learned model DB2 are separately generated, for example, when the scene is an existing scene and only content is added, learning related to the scene may not be performed.

[0142] In the present embodiment, the case of using a plurality of second learning completed model DB2s is described, but the present invention is not limited thereto, and the second learning completed model DB2 to be used may be one. In addition, in the present embodiment, the case of displaying an image of an object to be handled that can be inferred to be originally required is described, but the present invention is not limited thereto, and a part of an object to be handled that can be inferred to be originally required may also be displayed. In addition, in the present embodiment, an object to be handled that can be inferred to be originally unnecessary or a part of an object to be handled may also be prompted.

[0143] In the utilization phase, the information processing device 1 of this embodiment can also determine deficiencies by comparing the tree structure associated by the image segmentation unit and the hierarchical structure composed of values ​​output from the first learned model DB1 and the second learned model DB2.

[0144] Description of symbols

[0145] 1: information processing device; 2: central processing unit; 3: main storage device; 4: image acquisition unit; 5: image segmentation unit; 6: scene estimation unit; 7: data block estimation unit; 8: data block output unit; 9: first learning completion model generation unit; 10: second learning completion model generation unit; 11: auxiliary storage device; 12: user terminal; 13: recommended image output unit.

Claims

1. An information processing device that outputs information related to a task performed by a respondent, namely, task information, wherein: The information processing device comprises: an image acquisition unit that acquires, via a user terminal, an original image that includes a respondent responded to by the responder and a plurality of responded objects responded to by the responder; An image segmentation unit estimates the size of the object based on the information that does not change, and segments the original image into an object image of a size that includes the object and a plurality of object images of sizes that include each of the objects; a scene estimation unit, which estimates an event performed by a responder, i.e., a scene responded to by the responder, using a first learned model, wherein the first learned model stores an association between the responder image and a scene ID uniquely representing the scene, and the scene estimation unit uses the scene ID as a search keyword to obtain a scene name list from a scene table, and sends the obtained scene name list to the user terminal, wherein the scene table is a table in which the scene ID and the name of the scene, i.e., the scene name, are associated one-to-one; a data block estimation unit that uses a scene ID corresponding to a scene name selected by a user via the user terminal as a search keyword to retrieve a model ID, estimates a data block of the object by estimating a data block summary using one of a plurality of second learned models, and sends the estimated plurality of data block summaries to the user terminal, wherein the plurality of second learned models store associations between the plurality of object images and one or more data block meta IDs that correspond one-to-one to data block IDs that uniquely represent the data block that is information for dividing or suggesting the operation information; and an output unit that acquires a corresponding data block based on a data block ID corresponding to a data block digest received from the user terminal, and outputs the acquired data block to the user terminal, The data block estimation unit selects one of the plurality of second learned models using a model ID corresponding one-to-one to a scene ID, and the data block meta ID uniquely represents a data block meta value that is information related to the property of the object being addressed.

2. The information processing device according to claim 1, wherein: The information processing device may also have a recommended image output unit, which uses a combination of a model ID and one or more data blocks with a meta ID as a search keyword to retrieve a recommended target object image, wherein the recommended target object image is an image of the target object that is inferred to be originally required but is not captured in the original image.

3. The information processing device according to claim 1, wherein: The image segmentation unit associates the respondent with the plurality of respondent objects in a tree structure in which the respondent is a root node and the plurality of respondent objects are leaf nodes or internal nodes.

4. The information processing device according to claim 3, wherein: The image segmentation unit further acquires information included in at least one of the objects to be handled, and associates the information with the tree structure as a leaf node.

5. An information processing method, which is performed by an information processing device that outputs information related to a task performed by a respondent, namely, task information, wherein: The information processing method comprises the following steps: In the first step, an original image is obtained through a user terminal, which is an image including a respondent responded to by the responder and a plurality of responded objects responded to by the responder; The second step is to estimate the size of the object based on the information that does not change, and to divide the original image into an image of the object ... Step 3, using the first learning completion model to estimate the event performed by the responder, i.e., the scene responded by the responder, wherein the first learning completion model stores the association between the responder image and the scene ID uniquely representing the scene, using the scene ID as a search keyword, obtaining a list of scene names from a scene table, and sending the obtained list of scene names to the user terminal, wherein the scene table is a table in which the scene ID and the name of the scene, i.e., the scene name, are associated one-to-one; The fourth step is to retrieve a model ID using a scene ID corresponding to a scene name selected by a user via the user terminal as a search keyword, estimate a data block of the object by estimating a data block summary using one of a plurality of second learned models, and send the estimated plurality of data block summaries to the user terminal, wherein the plurality of second learned models store associations between the plurality of object images and one or more data block meta IDs, wherein the data block meta IDs correspond one-to-one to data block IDs uniquely representing the data blocks as information for dividing or suggesting the operation information; and Step 5, obtaining a corresponding data block based on a data block ID corresponding to a data block digest received from the user terminal, and outputting the obtained data block to the user terminal, One of the plurality of second learned models is selected using a model ID that corresponds one-to-one to a scenario ID, and the data block meta ID uniquely represents a data block meta value that is information related to the property of the object being handled.

Images