Program, method, information processing device, and system

Abstract

To provide a program, a method, an information processing apparatus, and a system which perform character recognition and proper processing on a recognized character without making a system configuration complicated, according to entry contents entered in a read object such as a business form.SOLUTION: The method includes: an acquisition step of acquiring business form data being entry contents included in a read target; a region detection step of applying an object detection model to the read target to detect one or more regions included in the read target, a class of each of the one or more regions, and a second character string being respective entry contents of the one or more regions; a processing specification step of specifying one or more pieces of processing according to respective classes of the one or more regions; and a processing application step of applying the processing according to the class of each of the one or more regions detected in the region detection step to a first character string acquired in the character string acquisition step by using the second character string in the one or more regions.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates to a program, a method, an information processing device, and a system. [Background technology]

[0002] Information processing devices that recognize characters are known. Patent Document 1 discloses an information processing device that, when editing the character recognition results of a character string containing multiple editing symbols in accordance with the editing symbols, can obtain character recognition results that are more accurately edited than when the device does not have this configuration. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-072593 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a problem in that it is not possible to perform character recognition and appropriate processing on the recognized characters according to the content written on the object to be read, such as a form, without requiring a complex system configuration. Therefore, the present disclosure has been made to solve the above-mentioned problems, and its purpose is to provide a technology that can flexibly perform character recognition and appropriate processing on the recognized characters with a simple system configuration. [Means for solving the problem]

[0005] A program to be executed by a computer having a processor and a storage unit, the program executing the following steps: a reading step in which the processor reads a read object having content written in a predetermined entry range; a character string acquisition step in which a first character string is acquired, which is the content written in the read object read in the read step; an area detection step in which an object detection model is applied to the read object read in the read step to detect one or more areas included in the read object, the classes of each of the one or more areas, and second character strings, which are the content written in each of the one or more areas; a processing identification step in which one or more processes are identified according to the classes of each of the one or more areas detected in the area detection step; and a processing application step in which a process according to the class of each of the one or more areas detected in the area detection step is applied to the first character string acquired in the character string acquisition step, using the second character strings in the one or more areas. [Effects of the Invention]

[0006] According to the present disclosure, it is possible to flexibly perform character recognition and appropriate processing on the recognized characters with a simple system configuration. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 2 is a block diagram showing the functional configuration of the system 1. [Figure 2] FIG. 2 is a block diagram showing the functional configuration of the server 10. [Figure 3] FIG. 2 is a block diagram showing the functional configuration of a user terminal 20. [Figure 4] FIG. 10 is a diagram showing the data structure of a user table 1012. [Figure 5] FIG. 10 is a diagram showing the data structure of a form table 1013. [Figure 6] FIG. 10 is a diagram showing the data structure of a detection table 1014. [Figure 7] FIG. 10 is a diagram showing the data structure of a processing table 1015. [Figure 8]10 is a flowchart showing the operation of a read process. [Figure 9] 10 is a flowchart showing the operation of a rewrite process. [Figure 10] 10 is a diagram illustrating an example of a read target showing the operation of a read process. [Figure 11] 10 is an example showing the operation of an area detection step in a rewriting process. [Figure 12] 10 is a screen example showing the operation of a processing result presentation step in the rewrite processing. [Figure 13] FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In all drawings describing the embodiments, common components are designated by the same reference numerals, and repeated description will be omitted. Note that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Furthermore, not all components shown in the embodiments are necessarily essential components of the present disclosure. Furthermore, each drawing is a schematic diagram and is not necessarily a precise illustration.

[0009] <System 1 Configuration> The system 1 in this disclosure is an information processing system that provides an image processing service. The system 1 includes an information processing device, a server 10, and a user terminal 20, which are connected via a network N. FIG. 1 is a block diagram showing the functional configuration of the system 1. As shown in FIG. FIG. 2 is a block diagram showing the functional configuration of the server 10. As shown in FIG. FIG. 3 is a block diagram showing the functional configuration of the user terminal 20. As shown in FIG.

[0010] Each information processing device is configured by a computer equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the server 10 and the user terminal 20, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer will be omitted.

[0011] <Server 10 configuration> The server 10 is an information processing device that provides image processing services. The server 10 includes a storage unit 101 and a control unit 104 .

[0012] <Configuration of the storage unit 101 of the server 10> The storage unit 101 of the server 10 includes an application program 1011 , an object detection model 1021 , a user table 1012 , a form table 1013 , a detection table 1014 , and a processing table 1015 .

[0013] The application program 1011 is a program for causing the control unit 104 of the server 10 to function as each functional unit. Application programs 1011 include applications such as a web browser application.

[0014] The object detection model 1021 is a learning model that extracts one or more image regions (regions) containing objects from input image data and outputs the class names of the objects contained in the regions. Specifically, it is a learning model that detects the presence of objects from input image data, specifies the region (bounding box) surrounding each object, identifies the class (category) to which the object belongs, and outputs the region and class name.

[0015] User table 1012 is a table that stores and manages information about member users (hereinafter, "users") who use the service. When a user registers to use the service, the user's information is stored in a new record in user table 1012. This allows the user to use the service according to the present disclosure. The user table 1012 is a table having columns of user ID, user name, and user data, with the user ID as the primary key. FIG. 4 is a diagram showing the data structure of the user table 1012. As shown in FIG.

[0016] The user ID is an item that stores user identification information for identifying a user. The user identification information is an item that is set with a unique value for each user. The user name is an item for storing the name of the user. The user name may be set to any character string such as a nickname instead of a name. The user data includes information specific to the individual user and attribute information relating to the user's characteristics and background. The user's unique information includes information unique to the user, such as the user's date of birth (age), sex, and the like. The user's attribute information includes information such as the user's educational history (highest level of education, major, graduation year), occupation, work history, interests, place of residence, and language.

[0017] The form table 1013 is a table for storing and managing information related to forms (form information). The form table 1013 is a table having a form ID as a primary key and columns of a form ID, form data, a first character string, and result data. FIG. 5 is a diagram showing the data structure of the form table 1013.

[0018] The form ID is an item that stores form identification information for identifying a form. The form identification information is an item that has a unique value set for each piece of form information. The form data is data related to various application documents and procedural documents. For example, the document data includes applications for opening a bank account, credit card applications, change of address notifications, driver's license renewal applications, passport applications, requests for copies of resident registration certificates, tax returns, company incorporation registration applications, and various license and permit applications. In this disclosure, this refers to various application documents, procedural documents, etc. printed on paper or the like, which have been manually filled out and then read by an image reading device such as a camera or scanner. Note that this may also include data manually filled out by hand on various application documents, procedural documents, etc. displayed on a tablet or the like. The first character string is text data obtained by reading the form data using OCR (optical character recognition) technology. The first character string includes text data obtained by applying the form data to the OCR unit 1052 as input data. The first character string may include structured text data such as JSON obtained by inputting the form data to the generation AI 50 as input data. The first character string may also include structured text data such as JSON obtained by directly inputting the form data to the generation AI 50 as input data. The first character string may also be text data obtained by applying both the OCR unit 1052 and the generation AI 50 to the form data. The result data is an item that stores the processing results of executing processing on the form data based on the processing data in the processing table 1015 corresponding to one or more detection targets detected from the form data. Details will be explained in the reading processing and rewriting processing sections.

[0019] The detection table 1014 is a table for storing and managing information (detection information) about the detection target (detected object) detected by the object detection unit 1051. The detection table 1014 is a table having the detection ID as a primary key and columns of the detection ID, form ID, class name, detection area, and second character string. FIG. 6 is a diagram showing the data structure of the detection table 1014. As shown in FIG.

[0020] The detection ID is an item for storing detection identification information for identifying a detection target. The detection identification information is an item for which a unique value is set for each detection target. The form ID is an item for storing form identification information for identifying a form. The class name is an item for storing the class name (category) of the detection target. In the present disclosure, classes related to the selection of "strikethrough", "insertion", "correction", and "selection" are disclosed as examples. The detection area is an area for identifying an area including a detection target in form data identified by a form ID. Specifically, if the detection area is rectangular, it includes multiple coordinate values ​​(e.g., coordinate values ​​of the vertices of the rectangle) for identifying the rectangle in the form data. The detection area may also include image data included in the detection area in the form data. For example, a portion of image data included in an area of ​​the form data that overlaps the detection area may be extracted, and the image data may be included in the detection area. The second character string is text data obtained by reading the detection area using OCR (Optical Character Recognition) technology. The second character string includes text data obtained by applying the detection area as input data to the OCR unit 1052. The second character string may also include structured text data such as JSON obtained by applying the detection area as input data to the generation AI 50. The second character string may also be text data obtained by applying both the OCR unit 1052 and the generation AI 50 to the detection area.

[0021] The processing table 1015 is a table for storing and managing information related to processing (processing information). The process table 1015 is a table having the process ID as a primary key, and columns of the process ID, class name, and process data. FIG. 7 is a diagram showing the data structure of the processing table 1015.

[0022] The process ID is an item for storing process identification information for identifying a process, and a unique value is set for each piece of process information. The class name is an item for storing the class name. In the present disclosure, classes related to the selection of "strikethrough", "insertion", "correction", and "selection" are disclosed as examples. The processing data is an item that stores processing details corresponding to the class name. Specifically, the processing data is an item that stores a program, command, function (method), etc. for executing a predetermined process on the second character string using the first character string and the second character string as arguments.

[0023] <Configuration of the control unit 104 of the server 10> The control unit 104 of the server 10 includes a user registration control unit 1041, an object detection unit 1051, and an OCR unit 1052. The control unit 104 executes an application program 1011 stored in the storage unit 101, thereby realizing each functional unit.

[0024] The user registration control unit 1041 performs processing to store information about users who wish to use the service according to the present disclosure in the user table 1012. The information stored in the user table 1012 is generated when a user opens a web page operated by a service provider from any information processing terminal, enters information into a predetermined input form, and transmits the information to the server 10. The user registration control unit 1041 stores the received information in a new record in the user table 1012, completing the user registration. This allows the user stored in the user table 1012 to use the service. Before the user registration control unit 1041 registers the user information in the user table 1012, the service provider may conduct a predetermined examination to restrict whether or not the user is permitted to use the service. The user ID may be any character string or number that can identify the user, any character string or number desired by the user, or may be automatically set by the user registration control unit 1041.

[0025] The object detection unit 1051 is a control unit that detects one or more objects included in image data using an object detection model.

[0026] The OCR unit 1052 is a control unit that applies OCR (Optical Character Recognition) technology to image data.

[0027] <Configuration of User Terminal 20> The user terminal 20 is an information processing device operated by a user who uses a service. The user terminal 20 may be, for example, a mobile terminal such as a smartphone or tablet, a stationary personal computer (PC) or a laptop PC, or a wearable terminal such as a head mounted display (HMD) or a wristwatch terminal. The user terminal 20 includes a storage unit 201 , a control unit 204 , an input device 206 , and an output device 208 .

[0028] <Configuration of the storage unit 201 of the user terminal 20> The storage unit 201 of the user terminal 20 includes a user ID 2011 and an application program 2012 .

[0029] The user ID 2011 is the user's account ID. The user transmits the user ID 2011 from the user terminal 20 to the server 10. The server 10 identifies the user based on the user ID 2011 and provides the user with the service according to the present disclosure. The user ID 2011 includes information such as a session ID temporarily assigned by the server 10 to identify the user using the user terminal 20.

[0030] The application program 2012 may be stored in advance in the storage unit 201, or may be configured to be downloaded from a web server or the like operated by a service provider via a communication IF. The application programs 2012 include applications such as a web browser application. The application program 2012 includes an interpreted programming language such as JavaScript (registered trademark) that runs on a web browser application stored on the user terminal 20.

[0031] <Configuration of the control unit 204 of the user terminal 20> The control unit 204 of the user terminal 20 includes an input control unit 2041 and an output control unit 2042. The control unit 204 executes an application program 2012 stored in the storage unit 201, thereby realizing each functional unit.

[0032] <Configuration of the input device 206 of the user terminal 20> The input device 206 of the user terminal 20 includes a camera 2061 , a microphone 2062 , a position information sensor 2063 , a motion sensor 2064 , and a touch device 2065 .

[0033] <Configuration of the output device 208 of the user terminal 20> The output device 208 of the user terminal 20 includes a display 2081 and a speaker 2082 .

[0034] <System 1 Operation> Each process of the system 1 will be explained below. FIG. 8 is a flowchart showing the operation of the read process. FIG. 9 is a flowchart showing the operation of the rewrite process. FIG. 10 shows an example of a read target illustrating the operation of the read process. FIG. 11 shows an example of the operation of the area detection step in the rewriting process. FIG. 12 is an example of a screen showing the operation of the processing result presentation step in the rewrite processing.

[0035] <Read process> The reading process is a process of reading various handwritten application forms and procedural documents as data.

[0036] <Outline of the read process> The reading process is a series of processes that reads documents containing written information, applies OCR (optical character recognition) to the documents to read text data related to the written information, generates structured data such as JSON or markdown from the read text data, and stores the text data and structured data in association with the documents.

[0037] <Read process details> The reading process will be described in detail below.

[0038] In step S101, the control unit 104 of the server 10 executes a read step of reading a read object in which entry content has been entered in a predetermined entry range. Specifically, the user operates the input device 206 of the user terminal 20 to execute a browser application or the like, and opens the read page by inputting the URL of a web page (read page) for executing the read process. The control unit 204 of the user terminal 20 transmits a request including the user ID 2011 for opening the read page to the server 10.

[0039] When the server 10 receives the request, it generates a read page and transmits it to the user terminal 20. The control unit 204 of the user terminal 20 displays and presents the read page on the display 2081 of the user terminal 20. The read page includes an upload button for uploading a predetermined file, etc. The user operates the input device 206 of the user terminal 20, takes a photograph of the object to be read, such as a paper form, using the camera 2061, and uploads the received image data from the reading page. By operating the camera 2061 of the user terminal 20, the user can read the contents written on any medium, such as a paper form, paper, whiteboard, or blackboard, as image data. In this disclosure, an example of reading a paper form, such as a bank account opening application form, on which a user fills out a form by hand will be described. However, the application of this disclosure is not limited to paper forms. For example, the disclosure can also be applied to image data such as handwritten input by a user using a touch panel of a tablet, smartphone, or the like.

[0040] Figure 11 shows an example of a read target that illustrates the operation of the read process. Specifically, form data D1 has fields for name, address, gender, date of birth, and phone number. These fields are written or printed as follows: Yamada Taro, 1-2-3 Shinjuku, Minami-Shinjuku-ku, Tokyo, Male / Female, YYYY-MM-MM, 00-0000-0000, respectively. The "Male / Female" field is a selectable field and is therefore pre-printed on the form, but the other fields may be pre-printed or may be handwritten. In the address field, the character "Minami" (South) in Minami-Shinjuku has been struck through D11 (cancelled). Also, between "Shinjuku" and "1-2-3" the character "Nishi" (West) D12 is written with a speech bubble (inserted). In the gender field, the character "Woman" is surrounded by a circle D13 (selection of an option). Of the phone number 0000, the two middle digits have been struck through D14 and the character "22" D15 has been written (corrected).

[0041] The control unit 204 of the user terminal 20 transmits the read image data to the server 10. The control unit 104 of the server 10 receives and accepts the image data received from the user terminal 20. The control unit 104 of the server 10 stores the received image data in the form data item of a new record in the form table 1013.

[0042] In step S102, the control unit 104 of the server 10 executes a character string obtaining step of obtaining a first character string that is the entry content included in the read target read in the read step. Specifically, the OCR unit 1052 of the server 10 applies OCR (optical character recognition technology) to the image data received in step S101, thereby converting character information included in the image data into text data (first character string).

[0043] In step S103, a character string acquisition step executes a step of acquiring, as a first character string, structured data output by inputting a read target into a large-scale language model. Specifically, the control unit 104 of the server 10 inputs the image data received in step S101, the text data output by the OCR unit 1052, and the like to the generation AI 50. For example, a request including the image data received in step S101, text data output by the OCR unit 1052, etc., along with instructions (prompts) for outputting the entry content included in the image data as structured data (JSON, markdown, etc.) based on characters, graphics, etc. included in the image data, is sent to a URL associated with an API (Application Programming Interface) endpoint provided by the generation AI 50. The generation AI 50 sends a response to the request to the server 10. The server 10 receives the response.

[0044] An example of structured data included in a response received by the server 10 from the generated AI 50 in response to image data scanned by the user terminal 20, such as a bank account opening application form, is shown below. [Example of structured data] { Name: Taro Yamada, Address: 1-2-3 Shinjuku, Minami-Shinjuku, Tokyo Gender: Male / Female, Date of Birth: YYYY-MM-MM, Phone number: 00-0000-0000 }

[0045] In step S103, a character string obtaining step is executed to obtain a plurality of first character strings as selection candidates. Specifically, the control unit 104 of the server 10 transmits a prompt including instructions for recognizing multiple character strings included in the first character string as selection candidates, along with the image data received in step S101, the text data output by the OCR unit 1052, and the structured data output from the generation AI 50, to an API endpoint provided by the generation AI 50. The generation AI 50 transmits a response to the request to the server 10. The server 10 receives the response. An example of structured data included in a response received by the server 10 from the generated AI 50 in response to image data scanned by the user terminal 20, such as a bank account opening application form, is shown below. In this case, the gender item can be recognized as a value relating to a selection candidate having the values ​​male and female, and acquired as structured data. [Example of structured data] { Name: Taro Yamada, Address: 1-2-3 Shinjuku, Minami-Shinjuku, Tokyo Gender: [ { Value: Male}, { Value: Female} ], Date of Birth: YYYY-MM-MM, Phone number: 00-0000-0000 }

[0046] In this way, by using OCR and large-scale language models, options (selection candidates) such as the select component of a web form, which are difficult to express on paper forms, can be obtained as structured data from image data read from documents such as application forms.

[0047] In step S104, the control unit 104 of the server 10 stores the text data and structured data acquired in steps S102 and S103 in the first character string field of the target record in the form table 1013. As a result, the control unit 104 of the server 10 associates the image data acquired in step S101, the text data acquired in step S102, and the structured data acquired in step S103, and stores them in the form data and first character string fields of the form table 1013. In the present disclosure, the first character string includes both text data of characters written in image data read by OCR and structured data obtained by using generated AI50 or the like.

[0048] <Rewrite process> The rewriting process is a process for acquiring the processing results by executing various processes on the read form data according to the detected detection target.

[0049] <Overview of rewriting process> The rewriting process is a series of processes that acquires the form data stored by the read process, detects one or more areas by applying an object detection model to the form data, applies OCR (optical character recognition) to the detected one or more areas, identifies a process for the one or more areas according to the class of the detected area, applies the process to the form data, stores the process results, and presents the process results.

[0050] The rewrite process may be executed following the read process already described, or may be configured to be executed at any timing different from the read process. In addition, the rewrite process may be executed synchronously with the read process or asynchronously.

[0051] <Details of the rewriting process> The rewriting process will be described in detail below.

[0052] <Report data acquisition steps> In step S301, the control unit 104 of the server 10 executes an acquisition step of acquiring form data. Specifically, the control unit 104 of the server 10 searches for records for which nothing is stored in the result data field of the form table 1013. Alternatively, the control unit 104 of the server 10 may refer to a rewrite processing flag (not shown) in the form table 1013, etc., to identify and acquire records for which the rewrite processing has not been executed (unexecuted). The control unit 104 of the server 10 acquires each field of the form data and first character string stored in the form table 1013.

[0053] <Area detection step> In step S302, the control unit 104 of the server 10 executes an area detection step in which an object detection model is applied to the read object read in the read step, thereby detecting one or more areas included in the read object, the class of each of the one or more areas, and a second character string which is the content written in each of the one or more areas. Specifically, the object detection unit 1051 of the server 10 inputs the form data (image data) acquired in step S301 as input data to the object detection model 1021. The object detection model 1021 outputs the regions of one or more objects included in the form data and the classes of the one or more regions. The object detection unit 1051 of the server 10 accepts the one or more regions and the classes of the one or more regions output from the object detection model 1021. The OCR unit 1052 of the server 10 converts character strings included in the detection regions into text data (second character strings) by applying OCR to detected images (image data) included in each detection region of the image data. Note that the object detection model 1021 may be configured to directly output text data related to the second character string together with the class and the region. The object detection model 1021 may also output the field name (item name) of the first character string to which processing is applied.

[0054] 11 shows an example of the operation of the object detection process in the rewrite process. From the form data D1, a strikethrough object D21, an insertion object D22, a selection item object D23, and a correction object D24 are detected by the object detection model 1021 of the server 10. In addition, which field (item) each detection area relates to can be identified by detecting the address field D31, gender field D32, and telephone number field D33 and determining the overlapping relationship between the area of ​​each field and the field of the detection area. Additionally, each detected object and the field (item name) corresponding to each object can be identified by the object detection model 1021. Note that the item name may be identified by inputting the detection area, etc., to the generation AI 50, etc. In the present disclosure, the strikethrough object D21 and the insert object D22 correspond to the address field, the selection object D23 corresponds to the gender field, and the correction object D24 corresponds to the phone number field.

[0055] An example of an object detected by the object detection model 1021 of the server 10 is shown below. The coordinates [x11, y11, x12, y12] of the area indicate the upper left and lower right coordinates of a rectangular detection area in the image data. The detection area does not necessarily have to be rectangular, and information indicating a specific range in the image data is stored in the area. The detection area does not necessarily have to be one, and may be specified as a combination of multiple areas. Furthermore, the object detection model 1021 can detect one or more objects. Specifically, the data output by the object detection model 1021 is data in a list format consisting of the following multiple objects.

[0056] Based on the detection areas of the strikethrough objects D21 and D31 in FIG. 11, information including the following second character string is extracted. "The character string including the part to be cancelled and the parts before and after it," "The part to be cancelled and the character string before it," "The part to be cancelled and the character string after it," "The part to be cancelled and the character string after it," "The part to be cancelled and the character string after it," "Cancellation character," and "The part to be cancelled and the item name" are class names (keys) output by the object detection model 1021 for the strikethrough object D21 and the area D31, respectively, and the right side is the value for each class. By training the object detection model 1021 in advance using training data, it is possible to output a value (character string) for each item. { Class name: strikethrough, Area: [x11,y11,x12,y12], Second string: { String including the part to be cancelled and the part before and after: 1-2-3 Shinjuku, Minami-Shinjuku-ku, Tokyo, The part to be cancelled and the characters before it: Tokyo, The part to be cancelled and the following text: Shinjuku 1-2-3, Cancellation: South }, Item name: Address },

[0057] Based on the detection areas of the insertion objects D22 and D31 in FIG. 11, information including the following second character string is extracted. "The character string including the insertion point and the characters before and after it," "The insertion point and the character string before it," "The insertion point and the character string after it," "The inserted character," and "The insertion point and the item name" are class names (keys) output by the object detection model 1021 for the inserted object D22 and the area D31, respectively, and the right side is the value for each class. By training the object detection model 1021 in advance using training data, it is possible to output a value (character string) for each item. { class name : infix expression, Area: [x21,y21,x22,y22], Second string: { The string containing the insertion point and its surroundings: 1-2-3 Shinjuku, Minami-Shinjuku-ku, Tokyo, Insertion point and the string before it: Shinjuku, Minami-Shinjuku, Tokyo, Insertion point and following string: 1-2-3, Inserted characters: West }, Item name: Address },

[0058] Based on the detection regions of the modified objects D24 and D33 in FIG. 11, information including the following second character string is extracted. "Character string including the correction point and the parts before and after it," "Correction point and the character string before it," "Correction point and the character string after it," "Cancellation character," "Insertion character," and "Cancellation point and item name" are class names (keys) output by object detection model 1021 for corrected object D24 and region D33, respectively, and the right side is a value for each class. By training object detection model 1021 in advance using training data, it is possible to output a value (character string) for each item. { Class name: Fix, Area: [x31,y31,x32,y32], Second string: { The string containing the part to be undone and its surroundings: 00-0000-0000, The part to be undone and the string before it: 00-0, The void and the following characters: 0-0000, Cancel character :00, Insert character :22, }, Item name: Phone number, },

[0059] Based on the detection areas of the selection item objects D23 and D32 in FIG. 11, information including the following second character string is extracted. "Selection candidate names 1 and 2" and "Selected candidate name" are class names (keys) output by the object detection model 1021 for the selection item object D23 and the area D32, respectively, and the right side is the value for each class. By training the object detection model 1021 in advance using training data, it is possible to output a value (character string) for each item. { Class name: Selection, Area: [x31,y31,x32,y32], Second string: { Selection candidate name 1: Male, Selection candidate name 2: Female, Selected candidate name: Female, }, Item name: gender, }

[0060] <Processing specific steps> In step S303, the control unit 104 of the server 10 executes a processing specification step of specifying one or more processes according to the respective classes of one or more areas detected in the area detection step. The processing specification step executes a step of specifying one or more processes according to at least one or more classes of strikethrough, insertion, and correction. Specifically, the control unit 104 of the server 10 searches the class name item of the processing table 1015 based on the class name of one or more areas detected in step S103, and identifies and acquires the processing data.

[0061] The process specifying step executes a step of specifying a strikethrough process, an insertion process, a modification process, and a selection process of a selected item according to the strikethrough class, the insertion expression class, the modification class, and the selection class of a selected item, respectively. The strikethrough process is a process of deleting the strikeout portion from the first string based on "the string including the strikeout portion and the characters before and after it," "the strikeout portion and the characters before it," "the strikeout portion and the characters after it," "the strikeout portion and the characters after it," "the strikeout character," and "the strikeout portion and the item name." The insertion process is a process of inserting an insertion character into the insertion point of the first string based on "the string including the insertion point and the characters before and after it," "the insertion point and the string before it," "the insertion point and the string after it," "the insertion character," and "the insertion point and the item name." The correction process is a process of deleting the cancellation character at the correction point from the first string and inserting the insertion character at the correction point based on the ``string including the correction point and the characters before and after it,'' ``the correction point and the string before it,'' ``the correction point and the string after it,'' ``cancellation character,'' ``insertion character,'' and ``cancellation point and item name.'' The selection process for the selection item is a process for selecting the selection candidate corresponding to the "selected candidate name" among the "selection candidate names 1 and 2" for the first character string.

[0062] <Processing application step> In step S304, the control unit 104 of the server 10 executes a processing application step in which processing according to each class of one or more areas detected in the area detection step is applied to the first character string acquired in the character string acquisition step using the second character string of one or more areas. Specifically, the control unit 104 of the server 10 applies the process identified in step S303 to the first character string using the second character string. In the present disclosure, a strikethrough process is identified based on a strikethrough object D21. An insert process is identified based on an insert object D22. A selection process for a selected item is identified based on a selection item object D23. A correction process is identified based on a correction object D24. If multiple processes are specified, the processes may be applied in order or simultaneously, and priorities may be assigned according to the content of the processes.

[0063] In step S304, the process application step identifies a process application target in the first character string based on the item name of the structured data, and applies a process according to the class to the process application target. Specifically, in an example of structured data in the present disclosure, strikethrough and insertion processes are applied to the address field of the structured data, selection of selection items is applied to the gender field of the structured data, and correction is applied to the phone number field of the structured data. The applied field (item) can be configured to be specified for each detection target detected by the object detection model.

[0064] <Processing application step (strikethrough processing)> In step S303, the strikethrough processing is executed on the detected area for which the strikethrough processing is specified. In step S304, the process applying step includes a step of identifying the start position and end position of the first character string, and a step of deleting, from the first character string, a character string included in the range from the start position to the end position. Specifically, by applying a strikethrough to the first character string and deleting the second character string that overlaps the struck-out portion in the "Address" field, the following output result can be obtained for the address item. - String reflecting the inserted location: 1-2-3 Shinjuku, Shinjuku-ku, Tokyo

[0065] <Processing application step (insertion process)> In step S303, the insertion process is executed for the detection region for which the process related to the insertion process has been identified. In step S304, the process applying step includes the steps of identifying an insertion position of the first character string and inserting the second character string into the insertion position of the first character string. Specifically, by applying the insert process to the first character string and inserting the second character string into the insertion point of the "Address" field, the following output result can be obtained for the address item. - String reflecting the inserted location: 1-2-3 Shinjuku Nishi, Minami-Shinjuku-ku, Tokyo

[0066] <Processing application step (correction process)> In step S303, the correction process is executed for the detection region for which the correction process is specified. In step S304, the processing application step includes a step of identifying the start position and end position of the first string, a step of deleting from the first string any string that falls within the range from the start position to the end position, and a step of inserting the second string at the start position of the first string. Specifically, by applying a correction process to the first string, deleting the cancellation characters in the cancellation locations in the "phone number" field, and inserting insertion characters, the following output result can be obtained for the phone number item. - String reflecting the inserted part: 00-0220-0000

[0067] <Processing application step (selection processing of selected items)> In step S303, the selection process for the selection item is executed for the detection area for which the process for the selection process for the selection item has been specified. In step S304, the process applying step executes a step of applying a process to select at least one selection candidate from the plurality of selection candidates acquired in the character string acquiring step, based on the first character strings in one or more areas. Specifically, by applying the selection candidate selection process to the first string and replacing the value of the "gender" field with the selected candidate name instead of the selected candidate, the following output result can be obtained for the gender item: ·Gender: Female Note that, as described below, an output result specified by a value (flag) indicating whether each selection item is selectable may be acquired. ·Gender: [ { value: male , choice: False}, { value: female , choice: True} ], This makes it possible to properly read and process information that is difficult to express on paper forms, such as the selection of options. Options such as the select component of a web form, which are difficult to express on paper forms, can be realized on paper documents such as application forms.

[0068] <Processing application step (variation example)> In step S304, the process applying step performs a process applying step by inputting the first string, the second string of one or more regions, and a prompt including instructions regarding one or more processes identified in the process identifying step into the large-scale language model. Each of the strikethrough process, insertion process, correction process, and selection process of a selected item can be realized as an operation on a character string using any programming language, etc. In addition, by sending a prompt including the first character string (structured data) to the generation AI 50 in the instruction (second character string) related to the strikethrough process, insertion process, correction process, and selection process of a selected item described above, the processing result for the first character string can be obtained.

[0069] For example, the following prompt can be used to cause the generation AI 50 to execute processing on the first character string. Note that the "Strikethrough processing content," "Insert processing content," "Modification processing content," and "Selection processing content for selected item" contain second character strings consisting of the item names (keys) and values ​​of the respective processes already explained. In other words, a predetermined process can be applied to the first character string simply by creating a prompt that instructs the application of the process described in the second character string to the first character string. Note that the use of the generation AI 50 has the advantage that, when multiple operations are specified for one item (for example, in the present disclosure, two operations, a strikethrough operation and an insertion operation, are specified for an address), the first character string can be appropriately processed without implementing complex operations. The large-scale language model can appropriately process the first character string even when multiple conflicting operations are specified. 〔prompt〕 Please apply one or more of the following processes to the target data below and output the processing results. #Target data { Name: Taro Yamada, Address: 1-2-3 Shinjuku, Minami-Shinjuku, Tokyo Gender: [ { Value: Male}, { Value: Female} ], Date of Birth: YYYY-MM-MM, Phone number: 00-0000-0000 } #Process 1 [Strikethrough content (second character string)] #Process 2 [Insertion process content (second character string)] #Process 3 [Content of correction process (second character string)] #Process 4 [Contents of selection process for selected items (second character string)]

[0070] In step S305, the control unit 104 of the server 10 acquires the character string related to the processing result obtained by applying the processing in step S304. [Example of structured data (processing result)] { Name: Taro Yamada, Address: 1-2-3 Shinjuku Nishi, Shinjuku-ku, Tokyo Gender: [ { value: male , choice: False}, { value: female , choice: True} ], Date of Birth: YYYY-MM-MM, Phone number: 00-0220-0000 }

[0071] The control unit 104 of the server 10 stores the acquired processing results in the result data item of the target record in the form table 1013. As a result, the processing results are stored in association with the form read in the read process.

[0072] <Presenting the processing results> In step S306, the control unit 104 of the server 10 transmits the processing result stored in step S305 to the user terminal 20. The user terminal 20 displays and presents the processing result on the display 2081 of the user terminal 20. For example, the control unit 104 of the server 10 may present a predetermined web form filled out based on the processing result. This allows the contents handwritten on a form or the like to be appropriately reflected in the web form or the like.

[0073] In step S306, the control unit 104 of the server 10 executes a selection presentation step in which at least one selection candidate selected in the processing identification step from among the multiple first character strings acquired as selection candidates in the character string acquisition step is presented in a manner different from the other selection candidates. Specifically, for a character string related to an item selected from multiple selection candidates such as "gender" in the processing results, the control unit 104 of the server 10 presents the selected selection candidate in a "selected state" and presents the unselected selection candidate in an "unselected state." 12 is an example screen showing the operation of the processing result presentation step in the rewrite processing. The name, address, and phone number items included in the output result are entered into the name input field D51, address input field D52, and phone number D54, respectively. For the gender item, only the female item D53B whose "selection" state is "True" is presented in a selected state, and the male item D53A whose "selection" state is "False" is presented in an unselected state.

[0074] <Basic computer hardware configuration> 13 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 901, a main memory device 902, an auxiliary memory device 903, and a communication IF 991 (interface), which are electrically connected to one another by a communication bus 921.

[0075] The processor 901 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

[0076] The main memory device 902 is used to temporarily store programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0077] The auxiliary storage device 903 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0078] The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using wired or wireless communication standards. The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks (e.g., Wi-Fi (registered trademark)) that can connect to the Internet via a predetermined access point. In the case of a wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), and Bluetooth (registered trademark). In the case of a wired connection, the network also includes a direct connection using a USB (Universal Serial Bus) cable, etc.

[0079] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration across multiple computers 90 and interconnecting them via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0080] <Basic functional configuration of computer 90> The following describes the functional configuration of a computer realized by the basic hardware configuration (FIG. 13) of the computer 90. The computer includes at least the functional units of a control unit, a storage unit, and a communication unit.

[0081] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 interconnected via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0082] The control unit is realized by the processor 901 reading out various programs stored in the auxiliary storage device 903, expanding them in the main storage device 902, and executing processing in accordance with the programs. The control unit can realize functional units that perform various types of information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0083] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes programs stored in memory. In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions. If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0084] The storage unit is realized by a main storage device 902 and an auxiliary storage device 903. The storage unit stores data, various programs, and various databases. Furthermore, the processor 901 can allocate a storage area corresponding to the storage unit in the main storage device 902 or the auxiliary storage device 903 in accordance with the programs. Furthermore, the control unit can cause the processor 901 to execute processes for adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0085] A database refers to a relational database, which manages data sets called masters and tables in a tabular format structurally defined by rows and columns, by relating them to each other. In a database, a table is called a table, a master, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables and masters can be set and associated. Typically, each table and each master has a column set as a primary key to uniquely identify a record, but setting a primary key to a column is not essential. The control unit can cause the processor 901 to add, delete, or update records in specific tables and masters stored in the storage unit according to various programs. Furthermore, by storing data, various programs, and various databases in the storage unit, it can be considered that the information processing device and information processing system according to the present disclosure have been manufactured.

[0086] Note that the databases and masters in this disclosure may include any data structure in which information is structurally defined (such as a list, dictionary, associative array, or object). The data structure also includes data that can be considered as a data structure by combining data with functions, classes, methods, etc. written in any programming language.

[0087] The communication unit is realized by the communication IF 991. The communication unit realizes a function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 901 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0088] <Additional Notes> The matters described in the above embodiments will be supplemented below.

[0089] (Appendix 1) A program to be executed by a computer having a processor and a memory unit, the program executing: a reading step (S101) in which the processor reads a read object having content written in a predetermined entry range; a character string acquisition step (S102) in which a first character string is acquired, which is the content written in the read object read in the read step; an area detection step (S302) in which an object detection model is applied to the read object read in the read step to detect one or more areas included in the read object, the class of each of the one or more areas, and a second character string, which is the content written in each of the one or more areas; a processing identification step (S303) in which one or more processes are identified according to the class of each of the one or more areas detected in the area detection step; and a processing application step (S304) in which, to the first character string acquired in the character string acquisition step, a process according to the class of each of the one or more areas detected in the area detection step is applied to the first character string acquired in the character string acquisition step, using the second character string in the one or more areas. By using an object detection model, it is possible to read the content of entries, including strikethroughs, insertions, corrections, etc., written on objects such as forms, and to perform general-purpose processing of character strings, which has traditionally been done mainly by hand, using a computer. It can also properly read and process information that is difficult to express on paper forms, such as selection of options. In the present disclosure, by using an object detection model, appropriate processing can be generally achieved without using a complex system configuration for expressions such as strikethroughs, insertions, and corrections entered on forms, even when the expression content (single lines, double lines, speech bubbles, etc.) differs depending on the country or cultural area.

[0090] (Appendix 2) 2. The program according to claim 1, wherein the character string acquisition step (S103) is a step of acquiring a first character string, which is structured data output by inputting the read target into a large-scale language model. This makes it possible to read the content entered on documents such as application forms as structured data in JSON format, markdown format, etc., and apply processing using the second character string to the structured data. Structured data such as input data received from a web form can be read from information entered manually on paper. Since the read input data is structured, no complex conversion process is required when storing it in a database or the like.

[0091] (Appendix 3) The program according to appendix 2, wherein the processing application step (S304) identifies a processing application target in the first character string based on the item name of the structured data, and applies processing according to the class to the application target. This makes it possible to read structured data from information entered manually in writing, and to reflect the entered content, including strikethroughs, insertions, corrections, and other expressions, in each item of the read structured data.

[0092] (Appendix 4) 4. The program according to any one of appendices 1 to 3, wherein the processing application step (S304) is a step of applying processing by inputting the first character string, the second character string in one or more regions, and a prompt including instructions regarding one or more processes identified in the processing identification step into a large-scale language model. This allows for the application of complex processing such as insertion, deletion, and selection in a general-purpose and flexible manner by using a large-scale language model.

[0093] (Appendix 5) A program described in any one of Appendices 1 to 3, wherein the object detection model is a learning model capable of detecting at least one or more of the following entries: strikethrough, insertion, and correction; and the processing identification step (S303) is a step of identifying one or more processing according to at least one or more classes of strikethrough, insertion, and correction. This allows the contents of entries, including strikethroughs, insertions, corrections, etc., written on the object to be read, such as a document, to be read, and processing of character strings, which has traditionally been mainly done manually, can be performed in a general-purpose manner by a computer.

[0094] (Appendix 6) A program described in any one of Appendices 1 to 3, wherein the object detection model is a learning model capable of detecting the content entered regarding the selection of a selection item, and the processing identification step (S303) is a step of identifying a selection processing for a selection item according to the class of the selection of the selection item. This makes it possible to properly read and process information that is difficult to express on paper forms, such as the selection of options. Information about options, such as the select component of a web form, which is difficult to express on paper forms, can be read from documents such as application forms.

[0095] (Appendix 7) The program described in Appendix 1, wherein the class detected in the area detection step is strikethrough, the processing identification step (S303) is a step of identifying a strikethrough processing according to the class of the strikethrough, and the processing application step (S304) includes a step of identifying a start position and an end position of a cancellation string included in a first string, and a step of deleting, from the first string, the cancellation string included in the range from the start position to the end position of the cancellation string. This allows the computer to read the content, including strikethroughs, written on the object to be read, such as a document, and appropriately perform processing based on the strikethrough expression on character strings, which has traditionally been done mainly by hand.

[0096] (Appendix 8) The program described in Appendix 1, wherein the class detected in the area detection step is an insertion expression, the processing identification step (S303) is a step of identifying an insertion process according to the class of the insertion expression, and the processing application step (S304) includes a step of identifying an insertion position of a first string and a step of inserting a second string into the insertion position of the first string. This allows the computer to read the contents of entries, including insertions (e.g., speech bubbles) made into the object to be read, such as a form, and to appropriately perform processing based on the expression of insertions in character strings, which has traditionally been done mainly by hand. The second character string to be inserted is not limited to characters, and may include control characters related to style, such as line breaks.

[0097] (Appendix 9) 2. The program according to claim 1, wherein the class detected in the area detection step is modification, the process identification step (S303) is a step of identifying a modification process according to the class of modification, and the process application step (S304) includes the steps of identifying a start position and an end position of a cancellation string included in a first string, deleting from the first string the cancellation string included in the range from the start position to the end position of the cancellation string, and inserting a second string into the first string at the start position. This allows the computer to read the entered content, including corrections (the string to be replaced is written with a double line, also called a correction) entered into the object to be read, such as a document, and appropriately perform processing based on the expression of corrections to the string, which has traditionally been done mainly by hand.

[0098] (Appendix 10) The program described in Appendix 1, wherein the string acquisition step (S103) is a step of acquiring multiple first strings as selection candidates, and the processing application step (S304) is a step of applying processing to select at least one selection candidate from the multiple selection candidates acquired in the string acquisition step based on the first strings of one or more areas. This makes it possible to properly read and process information that is difficult to express on paper forms, such as the selection of options. Information about options, such as the select component of a web form, which is difficult to express on paper forms, can be read from documents such as application forms.

[0099] (Appendix 11) The program described in Appendix 10, wherein the processor executes a selection presentation step (S306) of presenting at least one selection candidate selected in the processing identification step from among a plurality of first character strings acquired as selection candidates in the character string acquisition step, in a manner different from other selection candidates. This makes it possible to properly read and process information that is difficult to express on paper forms, such as the selection of options. Information about options, such as the select component of a web form, which is difficult to express on paper forms, can be read from documents such as application forms.

[0100] (Appendix 12) A computer-implemented method comprising a processor and a memory, wherein the processor performs all of the steps performed in the invention according to any one of appendixes 1 to 11. By using an object detection model, it is possible to read the content of entries, including strikethroughs, insertions, corrections, etc., written on objects such as forms, and to perform general-purpose processing of character strings, which has traditionally been done mainly by hand, using a computer. It can also properly read and process information that is difficult to express on paper forms, such as selection of options. In the present disclosure, by using an object detection model, appropriate processing can be generally achieved without using a complex system configuration for expressions such as strikethroughs, insertions, and corrections entered on forms, even when the expression content (single lines, double lines, speech bubbles, etc.) differs depending on the country or cultural area.

[0101] (Appendix 13) An information processing device comprising a control unit and a memory unit, wherein the control unit executes all of the steps executed in the invention according to any one of Supplementary Note 1 to Supplementary Note 11. By using an object detection model, it is possible to read the content of entries, including strikethroughs, insertions, corrections, etc., written on objects such as forms, and to perform general-purpose processing of character strings, which has traditionally been done mainly by hand, using a computer. It can also properly read and process information that is difficult to express on paper forms, such as selection of options. In the present disclosure, by using an object detection model, appropriate processing can be generally achieved without using a complex system configuration for expressions such as strikethroughs, insertions, and corrections entered on forms, even when the expression content (single lines, double lines, speech bubbles, etc.) differs depending on the country or cultural area.

[0102] (Appendix 14) A system comprising means for performing all steps performed in any of the inventions according to any one of appendixes 1 to 11. By using an object detection model, it is possible to read the content of entries, including strikethroughs, insertions, corrections, etc., written on objects such as forms, and to perform general-purpose processing of character strings, which has traditionally been done mainly by hand, using a computer. It can also properly read and process information that is difficult to express on paper forms, such as selection of options. In the present disclosure, by using an object detection model, appropriate processing can be generally achieved without using a complex system configuration for expressions such as strikethroughs, insertions, and corrections entered on forms, even when the expression content (single lines, double lines, speech bubbles, etc.) differs depending on the country or cultural area. [Explanation of symbols]

[0103] 1 system, 10 server, 101 memory unit, 104 control unit, 106 input device, 108 output device, 20 user terminal, 201 memory unit, 204 control unit, 206 input device, 208 output device

Claims

1. A program to be executed by a computer having a processor and a storage unit, the processor: a reading step of reading a read target in which entry content has been entered in a predetermined entry range; a character string obtaining step of obtaining a first character string that is an entry included in the read target read in the reading step; an area detection step of detecting one or more areas included in the read object, the classes of each of the one or more areas, and a second character string that is the content written in each of the one or more areas by applying an object detection model to the read object read in the reading step; a process specifying step of specifying one or more processes according to the class of each of the one or more regions detected in the region detecting step; a processing application step of applying processing corresponding to each class of the one or more areas detected in the area detection step to the first character string acquired in the character string acquisition step, by using the second character string of the one or more areas; Run the process applying step is a step of applying the process by inputting the first character string, the second character string in the one or more regions, and a prompt including instructions regarding the one or more processes identified in the process identifying step into a large-scale language model. program.

2. The object detection model is a learning model capable of detecting at least one or more of strikethroughs, insertions, and corrections; the process identifying step is a step of identifying one or more processes according to at least one or more classes of strikethrough, insertion, and correction; The program according to claim 1.

3. the class detected in the area detection step is strikethrough; the process specifying step is a step of specifying a strikethrough process according to the class of the strikethrough, The treatment applying step includes: Identifying a start position and an end position of a cancellation string included in the first string; deleting the cancellation string from the first string, the cancellation string being within a range from a start position to an end position of the cancellation string; Including, The program according to claim 1.

4. the class detected in the region detection step is an infix expression, the process identification step is a step of identifying an insertion process according to a class of the insertion expression; The treatment applying step includes: identifying an insertion position for the first character string; inserting the second character string into the first character string at the insertion position; Including, The program according to claim 1.

5. the class detected in the region detection step is a modification; the process identification step is a step of identifying a correction process according to the class of the correction, The treatment applying step includes: Identifying a start position and an end position of a cancellation string included in the first string; deleting the cancellation string from the first string, the cancellation string being within a range from a start position to an end position of the cancellation string; inserting the second character string into the first character string at the starting position; Including, The program according to claim 1.

6. A method implemented on a computer having a processor and a memory, wherein the processor performs all of the steps performed in the invention according to any one of claims 1 to 5.

7. 6. An information processing device comprising a control unit and a storage unit, wherein the control unit executes all of the steps executed in the invention according to any one of claims 1 to 5.

8. A system comprising means for executing all steps performed in the invention according to any one of claims 1 to 5.

Images