Visual information recognition system, visual information processing device, and visual information processing method

Abstract

To provide a visual information recognition system that contributes to quality improvement of image data.SOLUTION: A visual information recognition system includes: an imaging device 40 that images a medium that displays visual information to generate image data on which the medium appears; and a visual information processing device 10. The imaging device 40 is configured to generate characteristic information indicating characteristics of the imaging device 40 when it images the medium and transmit the image data and the characteristic information to the visual information processing device 10. The visual information processing device 10 includes an information recognition unit 11 that recognizes the visual information from the image data and a storage unit 12 that stores the visual information and the characteristic information in association with each other.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a visual information recognition system, a visual information processing device, and a visual information processing method. [Background technology]

[0002] For example, in the visual information recognition system disclosed in Patent Document 1, a medium is imaged by an imaging device (referred to as an "imaging unit" in the document), and visual information written on the medium is obtained based on the image data of the image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-136168 Summary of the Invention [Problem to be solved by the invention]

[0004] Various devices, such as smartphones, can be used as imaging devices. However, the imaging performance and program behavior of the various devices differ from one another. This means that the performance and program behavior of the device capturing the image of the medium are directly linked to the quality of the image data. In particular, poor image data quality makes it difficult to properly acquire visual information. However, with a wide variety of models and models available, it is often difficult to determine which devices are suitable or unsuitable for capturing images of the medium.

[0005] An object of the present disclosure is to provide a visual information recognition system that contributes to improving the quality of image data. [Means for solving the problem]

[0006] The characteristic configuration of a visual information recognition system for achieving the above-mentioned object is that it comprises an imaging device that images a medium that displays visual information and generates image data of the medium, and a visual information processing device, wherein the imaging device is configured to generate characteristic information that indicates the characteristics of the imaging device when the medium is imaged and to transmit the image data and the characteristic information to the visual information processing device, and the visual information processing device is equipped with an information recognition unit that recognizes the visual information from the image data and a memory unit that stores the visual information in association with the characteristic information.

[0007] Another characteristic configuration of the visual information recognition system according to the present disclosure is that it is provided with a reception unit that receives corrections from an operator who confirms the accuracy of the visual information recognized by the information recognition unit, and the memory unit is configured to store correction information indicating the correction information received by the reception unit in association with the characteristic information.

[0008] Another characteristic configuration of the visual information recognition system according to the present disclosure is that the visual information processing device is provided with a performance information generation unit that generates recognition performance information indicating the accuracy of the visual information recognized from the image data, and the memory unit is configured to store the recognition performance information in association with the characteristic information.

[0009] Another characteristic configuration of the visual information recognition system according to the present disclosure is that the memory unit is configured to store the time when the imaging device captured the medium in association with the recognition performance information.

[0010] Another characteristic configuration of the visual information recognition system according to the present disclosure is that the medium is a form on which the sales amount of the store is recorded.

[0011] Another characteristic configuration of the visual information recognition system according to the present disclosure is that the medium is a gift certificate.

[0012] The characteristic configuration of a visual information processing device for achieving the above-mentioned object is that the visual information processing device acquires, from an imaging device that has imaged a medium displaying visual information, image data of the medium and characteristic information that indicates the characteristics of the imaging device when the imaging device imaged the medium, and is equipped with an information recognition unit that recognizes the visual information from the image data and a memory unit that stores the visual information in association with the characteristic information.

[0013] The characteristic configuration of the visual information processing method for achieving the above-mentioned object is that it comprises an imaging step of imaging a medium displaying visual information with an imaging device and generating image data of the medium, a characteristic information generation step of generating characteristic information indicating the characteristics of the imaging device when the medium was imaged in the imaging step, an information recognition step of recognizing the visual information from the image data, and a storage step of associating and storing the visual information and the characteristic information.

[0014] According to the above characteristic configuration, it is possible to provide a visual information recognition system that contributes to improving the quality of image data. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram illustrating an example of a visual information recognition system. [Figure 2] FIG. 10 is a flowchart showing the flow up to recognition of visual information. [Figure 3] FIG. 10 is a flowchart showing the flow of confirmation of correctness by the tenant side. [Figure 4] FIG. 10 is a flowchart showing the flow of correctness confirmation by the owner side. [Figure 5] FIG. 10 is a diagram illustrating the transmission of form information using a graphic code. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, a visual information recognition system according to the present disclosure will be described with reference to the accompanying drawings. Note that the present disclosure is not limited to the following embodiments, and various modifications are possible without departing from the spirit and scope of the present disclosure.

[0017] [Basic configuration of visual information recognition system] 1, the visual information recognition system includes a visual information recognition server 10 ("visual information processing device"), a teller machine 20, a scanner 30, a portable imaging terminal 40 ("imaging device") with a camera function, and a confirmation terminal 50. The visual information recognition server 10, the teller machine 20, the scanner 30, the portable imaging terminal 40, and the confirmation terminal 50 are connected to each other via a wide area network (WAN).

[0018] The cash register 20 and scanner 30 are located in the cash register of the commercial complex. The scanner 30 may also be located in a tenant store. The confirmation terminal 50 is located in the management office of the commercial complex. The confirmation terminal 50 may be a general-purpose PC (personal computer), a smartphone, or a tablet terminal. In addition, a commercial complex has multiple stores as tenants. In many cases, rent for tenants in a commercial complex is paid on a commission basis that varies depending on sales. In such cases, the tenant's daily sales are deposited into the cash register 20. After the owner (developer) accurately confirms the tenant's sales, the tenant receives the amount remaining after deducting the rent. In addition, the commercial complex may also be an amusement complex.

[0019] The cash register 20 counts the amount of sales proceeds and transmits the amount to the owner's sales management server (not shown). When a voucher is deposited in the cash register 20, the cash register 20 scans the voucher with an internal scanner and transmits image data of the voucher to the owner's sales management server. Alternatively, the cash register may be scanned by the scanner 30, and the image data may be transmitted from the scanner 30 to the owner's sales management server.

[0020] A tenant's daily sales are recorded on Form 1 ("media"), such as receipts, and Form 1 is output from a cash register (not shown) in a POS (point-of-sale) system. The tenant's POS system is connected online to the tenant's own sales management system, but is not connected to the owner's sales management server. Furthermore, due to factors such as a lack of standardized communication standards and the inability of the tenant and owner to reach an agreement regarding the leakage of trade secrets, it is often not possible to connect the tenant's POS system and the management system online. For this reason, the tenant must separately submit the contents of Form 1 as evidence of daily sales.

[0021] As one specific means of submitting evidence of daily sales, the tenant operator scans the form 1 with the scanner 30 and transmits the image data of the form 1 to the visual information recognition server 10. As another means of submitting evidence of daily sales, the tenant operator captures an image of the form 1 with the portable imaging terminal 40 and transmits the image data of the image of the form 1 to the visual information recognition server 10.

[0022] The portable imaging terminal 40 is, for example, a smartphone or a tablet terminal, and is equipped with, for example, a CMOS camera module. The tenant operator captures an image of the form 1 using the portable imaging terminal 40. The portable imaging terminal 40 generates image data of the image of the form 1 and transmits the image data to the visual information recognition server 10.

[0023] The visual information recognition server 10 includes an information recognition unit 11, a storage unit 12, and a performance information generation unit 13 (described later). The information recognition unit 11 has an OCR (optical character recognition) function that recognizes characters, graphic codes, and the like from image data and scanned image data. In other words, the information recognition unit 11 is configured to be able to recognize visual information such as characters and graphics from both image data and scanned image data. Therefore, the visual information recognition server 10 functions as a so-called OCR server.

[0024] Among visual information, the acquisition of text information in particular can be achieved using a so-called well-known OCR function and an AI-OCR function that utilizes artificial intelligence. For example, the OCR function can be used to acquire text information, and the AI-OCR function can be used to compare the acquired text information to determine whether there are any recognition errors.

[0025] The visual information is stored in the storage unit 12. The storage unit 12 includes a storage medium (for example, a hard disk or a flash memory). The storage unit 12 stores image data of the document 1, image data of the document 1 scanned by the scanner 30, and visual information of the document 1 recognized from the image data and the image data.

[0026] When the form 1 is imaged with the portable imaging terminal 40, the tenant operator checks whether the visual information recognized by the information recognition unit 11 matches the content written on the form 1 after the image is captured. If the visual information does not match the content written on the form 1, it is possible that the OCR function of the information recognition unit 11 has failed to recognize the visual information (misreading or non-reading). A misreading means that a character or number is mistakenly recognized as another character. A non-reading means that a character or number is not recognized. The tenant operator corrects the misreading or non-reading in the visual information using the portable imaging terminal 40. Then, when the tenant operator has completed checking the accuracy of the visual information, the tenant reports the completion to the owner.

[0027] When the document 1 is scanned by the scanner 30, the owner's staff member uses the confirmation terminal 50 to confirm whether the amount of sales money deposited in the cash register 20 matches the sales amount based on the visual information stored in the memory unit 12. Also, when the document 1 is imaged by the portable imaging terminal 40, the owner's staff member uses the confirmation terminal 50 to confirm whether the amount of sales money deposited in the cash register 20 matches the sales amount based on the visual information stored in the memory unit 12 after the tenant has completed the work of confirming the visual information.

[0028] If there is a discrepancy between the amount of sales money deposited in the cash register 20 and the sales amount based on the visual information, the owner's staff will check the image data of the document 1 and the image data of the document 1 on the display screen of the confirmation terminal 50 to check for misreadings or non-readings in the visual information and make corrections as necessary. In this case, the owner's staff will correct any misreadings or non-readings in the visual information while inquiring with the tenant as necessary. Checking for misreadings or non-readings in the visual information is called "correctness check."

[0029] [The process of recognizing visual information] The flow up to recognizing visual information will be described with reference to the flowchart in Fig. 2. The tenant operator selects a method for reading the form 1 (step #01). In this embodiment, the form 1 is read by either the scanner 30 or the portable imaging terminal 40.

[0030] When the form 1 is read by the scanner 30 (Step #01: Reading by Scanner), the tenant operator places the form 1 in a predetermined position on the scanner 30, and the scanner 30 scans the form 1 (Step #02). The scanner 30 generates characteristic information of the scanner 30 at the time of scanning (Step #03). The characteristic information of the scanner 30 includes, for example, the model, model number, resolution, brightness, gamma correction value, and contrast of the scanner 30. The scanner 30 then transmits the image data of the scanned form 1 together with the characteristic information of the scanner 30 to the visual information recognition server 10 (Step #04).

[0031] When the document 1 is read by the portable imaging terminal 40 (step #01: reading by portable imaging terminal), when the tenant operator operates the shutter button (including the touch button displayed on the display) while the document 1 is displayed on the display of the portable imaging terminal 40, the document 1 is imaged by the camera module of the portable imaging terminal 40 (step #05, imaging step).

[0032] When the processing of step #05 is completed, it is determined whether or not the imaging of the entire area of ​​the document 1 has been completed (step #06). For example, if the document 1 is a vertically long receipt that does not fit within the imaging range of the portable imaging terminal 40, the imaging of the document 1 by the portable imaging terminal 40 is divided into multiple times, for example, the upper, middle, and lower parts of the document 1. The process of determining whether or not the imaging of the entire area of ​​the document 1 has been completed is performed, for example, by a method of checking whether or not the outer frame of the document 1 exists (whether or not the document 1 is cut off) based on multiple still image data captured by the portable imaging terminal 40. Note that the entire area of ​​the document 1 is not limited to all areas of the document 1. The entire area of ​​the document 1 may be, for example, only the area in which visual information of the necessary items to be obtained is displayed.

[0033] If imaging of the entire area of ​​the form 1 has not been completed (step #06: No), the processing of step #05 described above is repeated. Note that the determination of step #06 may be configured so that a Yes determination is made when the operator performs an operation to complete imaging. With this configuration, for example, when the operator determines that imaging of the necessary areas of the form 1 has been completed, it becomes possible to manually proceed to step #07.

[0034] When capturing the entire form 1 is complete (Step #06: Yes), the portable imaging terminal 40 combines the multiple image data representing the form 1 into a single image representing the entire form 1 (Step #07). The portable imaging terminal 40 also generates characteristic information for the portable imaging terminal 40 at the time of capturing the image (Step #08, characteristic information generation step). The characteristic information for the portable imaging terminal 40 includes, for example, the model, model number, resolution, imaging magnification, F-number, whether the HDR (high dynamic range) function is enabled or disabled, brightness, gamma correction value, contrast, capture time, and the name, version, startup method, and program behavior of the imaging application. If the image capture was performed using an Internet browser application, the characteristic information may also include the name, model number, and version information of the Internet browser application. The portable imaging terminal 40 then transmits the single image data representing the entire form 1 together with the characteristic information for the portable imaging terminal 40 to the visual information recognition server 10 (Step #09). In other words, the portable imaging terminal 40 is configured to generate characteristic information indicating the characteristics of the portable imaging terminal 40 when the form 1 is imaged, and to transmit the image data and the characteristic information to the visual information recognition server 10.

[0035] The visual information recognition server 10 receives image data and characteristic information of the scanner 30 from the scanner 30, and receives image data and characteristic information of the portable imaging terminal 40 from the portable imaging terminal 40. At this time, the visual information recognition server 10 stores this characteristic information in the storage unit 12 (step #10). The information recognition unit 11 of the visual information recognition server 10 recognizes visual information from the data (received image data and image data) using an OCR function (step #11, information recognition step). The visual information recognition server 10 then associates the recognized visual information with characteristic information of the device and stores it in the storage unit 12 (step #12, storage step). Here, the device refers to the scanner 30 if the visual information is based on scanning by the scanner 30, and to the portable imaging terminal 40 if the visual information is based on imaging by the portable imaging terminal 40. As a result, the visual information recognized from each of the image data and image data is linked to the characteristic information of the scanner 30 and the portable imaging terminal 40.

[0036] [Tenant's confirmation of accuracy] Even when the information recognition unit 11 utilizes the AI-OCR function described above, it is not possible to completely prevent misreading or non-reading of characters. For this reason, the tenant operator or the like who captures the image of the form 1 with the portable imaging terminal 40 must confirm, after capturing the image, whether the visual information recognized by the information recognition unit 11 matches the actual content written on the form 1.

[0037] The tenant's confirmation of accuracy will be described with reference to FIG. 3. An operator or the like who has captured an image of the form 1 with the portable imaging terminal 40 uses the browser screen of the portable imaging terminal 40 to request the visual information recognition server 10 to transmit visual information (step #21). Specifically, when the operator or the like selects a desired date and time or icon from a list of past image data transmission dates and times on the browser screen of the portable imaging terminal 40 and touches an icon button such as "Read" or "Confirm," a request signal is transmitted from the portable imaging terminal 40 to the visual information recognition server 10. In response to the request from the portable imaging terminal 40, the visual information recognition server 10 transmits visual information to the portable imaging terminal 40 (step #22). Until then, the portable imaging terminal 40 repeatedly determines in step #23 (step #23: No) and waits to receive visual information from the visual information recognition server 10. After executing step #22, the visual information recognition server 10 enters a waiting state in step #28.

[0038] When the portable imaging terminal 40 receives the visual information from the visual information recognition server 10 (step #23: Yes), the visual information is displayed in a list by item on the display of the portable imaging terminal 40. The visual information of the form 1 includes multiple items such as sales amount, VAT (Value Added Tax, which may be Japanese consumption tax), total sales, sales by product category or service category, product price, service price, discount amount, amount received in cash, amount received in gift certificates, etc., amount received through cashless payment, etc. These items are categorized and displayed on the display of the portable imaging terminal 40.

[0039] The tenant operator or the like checks whether there are any items that the information recognition unit 11 failed to recognize as visual information (step #24). In this embodiment, the portable imaging terminal 40 is provided with a reception unit 41 (FIG. 1). The reception unit 41 receives corrections from the operator who checks whether the visual information recognized by the information recognition unit 11 is correct. For example, the reception unit 41 is a button or the like for operating a touch panel on the portable imaging terminal 40.

[0040] When the operator or the like finds an item that the information recognition unit 11 has failed to recognize (step #24: Yes), the operator or the like operates the reception unit 41 to correct the visual information of the target item (step #25). At this time, the portable imaging terminal 40 counts the number of corrected items (step #26). The routine of steps #24 to #26 is repeated until no more items that have failed to be recognized are found.

[0041] If there are no items that the information recognition unit 11 failed to recognize as visual information, or if there are no more items (step #24: No), the portable imaging terminal 40 transmits confirmation result data indicating that the confirmation work by the tenant operator or the like has been completed to the visual information recognition server 10 (step #27). The confirmation result data includes the corrected visual information, a history of the corrected items, and the number of corrected items. The history of the corrected items and the number of corrected items are correction information indicating the information of the corrections received by the reception unit 41.

[0042] After executing step #22, the visual information recognition server 10 determines whether or not confirmation result data has been received from the portable imaging terminal 40 (step #28). Until the processing of step #27 is executed in the portable imaging terminal 40, the visual information recognition server 10 waits for the completion of the confirmation work by an operator or the like.

[0043] When the visual information recognition server 10 receives the confirmation result data from the portable imaging terminal 40 (step #28: Yes), it stores the corrected (updated) visual information in the storage unit 12 (step #29). At this time, the storage unit 12 is configured to store correction information indicating the information of the correction received by the reception unit 41 in association with the characteristic information of the portable imaging terminal 40. Then, the performance information generation unit 13 of the visual information recognition server 10 generates recognition performance information based on the corrected (updated) visual information (step #30). In other words, the performance information generation unit 13 generates recognition performance information indicating the accuracy of the visual information recognized from the image data based on the correction information. At this time, the storage unit 12 is configured to store the recognition performance information in association with the characteristic information of the portable imaging terminal 40.

[0044] The recognition performance information may be, for example, the ratio of the number of corrected items to the total number of items, i.e., "Recognition performance information = number of corrected items / total number of items" The result information is quantified as "accuracy" and indicates the accuracy of the visual information recognized from the image data or the visual information recognized from the image data. The result information generation unit 13 stores the recognition result information in the storage unit 12. Then, the visual information recognition server 10 generates completion report data for the tenant (step #31). The completion report data is transmitted from the visual information recognition server 10 to the confirmation terminal 50.

[0045] [Check by the owner] After the confirmation terminal 50 receives the completion report data, the owner's operator or the like checks whether the visual information recognized by the information recognition unit 11 matches the actual content written on the form 1.

[0046] The visual information from the scanner 30 is checked for accuracy by an operator of the owner, etc., without being checked by the tenant.

[0047] The owner's confirmation of accuracy will be described with reference to FIG. 4. The owner's operator or the like uses the browser screen of the confirmation terminal 50 to request the visual information recognition server 10 to send visual information (step #41). Specifically, when the operator or the like selects the desired case from a list of forms linked to the completion report data on the browser screen of the confirmation terminal 50 and touches an icon button such as "Read" or "Confirm," a request signal is sent from the confirmation terminal 50 to the visual information recognition server 10. In response to the request from the confirmation terminal 50, the visual information recognition server 10 sends visual information and image data or image data to the confirmation terminal 50 (step #42). Until then, the confirmation terminal 50 repeatedly determines in step #43 (step #43: No) and waits to receive data from the visual information recognition server 10. After executing step #42, the visual information recognition server 10 enters a waiting state in step #50.

[0048] When the confirmation terminal 50 receives the visual information from the visual information recognition server 10 (step #43: Yes), the visual information is displayed in a list by item on the display of the confirmation terminal 50. The owner's operator or the like checks whether all the necessary items are present (step #44). In this embodiment, the confirmation terminal 50 is provided with a reception unit 51. The reception unit 51 accepts corrections from the operator who checks whether the visual information recognized by the information recognition unit 11 is correct. For example, the reception unit 51 is a touch panel operation button, a mouse, a keyboard, etc. on the confirmation terminal 50.

[0049] Reports from tenants who have recently moved in or opened, or from tenants where the person in charge has not properly handed over their duties, may not include all the necessary items. Therefore, if the necessary items are not included (step #44: No), the owner's operator or other person will inquire with the tenant's person in charge by phone or email (step #45). When the owner's operator or other person obtains information about the items from the tenant's person in charge, they operate the reception unit 51 of the confirmation terminal 50 to input the necessary information (visual information) about the missing items (step #47). At this time, the confirmation terminal 50 counts the number of corrected items (step #48).

[0050] The owner's operator or the like checks whether there are any items that the scanner 30 failed to recognize as visual information or that the tenant's operator or the like overlooked due to misreading or non-reading (step #46). If the operator or the like finds an item that failed to be recognized (step #46: Yes), the operator or the like operates the reception unit 51 to correct the visual information for the target item (step #47). At this time, the confirmation terminal 50 counts the number of corrected items (step #48). The routine of steps #44 to #48 is repeated until no more defective items are found.

[0051] If there are no errors in the visual information or if there are no errors (step #44: Yes and step #46: No), the confirmation terminal 50 transmits confirmation result data indicating that the confirmation work by the owner's operator or the like has been completed to the visual information recognition server 10 (step #49). The confirmation result data includes the corrected visual information, a history of the corrected items, and the number of corrected items. The history of the corrected items and the number of corrected items are correction information indicating the information on the corrections received by the reception unit 51.

[0052] After executing step #42, the visual information recognition server 10 determines whether or not confirmation result data has been received from the confirmation terminal 50 (step #50). Until the processing of step #49 is executed in the confirmation terminal 50, the visual information recognition server 10 waits for the completion of the confirmation work by an operator or the like.

[0053] When the visual information recognition server 10 receives the confirmation result data from the confirmation terminal 50 (Step #50: Yes), it stores the corrected (updated) visual information as confirmed visual information in the memory unit 12 (Step #51). At this time, the memory unit 12 is configured to store correction information indicating the correction information received by the reception unit 51 in association with characteristic information of the device. Here, the device refers to the scanner 30 if the visual information is based on scanning by the scanner 30, or the portable imaging terminal 40 if the visual information is based on imaging by the portable imaging terminal 40. Then, the performance information generation unit 13 of the visual information recognition server 10 updates the recognition performance information stored in the memory unit 12 based on the correction information received from the confirmation terminal 50 (Step #52). At this time, the memory unit 12 is configured to store the recognition performance information in association with characteristic information of the device.

[0054] The recognition performance information can be checked, for example, by an operator of the owner or the like on the confirmation terminal 50, or by an operator of the tenant or the like on the portable imaging terminal 40. The fewer the number of corrected items, the better the recognition performance information, and the more the number of corrected items, the worse the performance. The recognition performance information is generated based on the characteristic information of the portable imaging terminal 40. This makes it possible to understand the correlation between the performance and imaging settings of the camera module installed in the portable imaging terminal 40 and the frequency of misreading or non-reading of visual information by the information recognition unit 11. This makes it possible to analyze terminals that frequently cause misreading or non-reading, as well as imaging settings and imaging environments (including camera shake during imaging and the degree of tilt of the document 1) that frequently cause misreading or non-reading, which can contribute to improving the accuracy of visual information recognition by OCR.

[0055] The characteristic information of the portable imaging terminal 40 also includes the imaging time when the portable imaging terminal 40 captured the image of the form 1. The storage unit 12 is configured to store the time when the portable imaging terminal 40 captured the image of the form 1 in association with the recognition performance information. For example, if the recognition performance information deteriorates at night, it is possible that the image of the form 1 was captured in a dark place. For this reason, the owner can urge the tenant to capture the image in a bright place, change the imaging settings to be more resistant to darkness, or use a portable imaging terminal 40 with excellent noise removal performance.

[0056] The memory unit 12 also stores recognition performance information associated with the visual information recognized by the information recognition unit 11 from the data scanned by the scanner 30. Therefore, for example, an operator of the owner can evaluate the recognition performance information based on the portable imaging terminal 40 based on the recognition performance information based on the scanner 30.

[0057] [Transmitting form information using graphic codes] As mentioned above, OCR technology cannot completely prevent the risk of misreading or non-reading. On the other hand, as shown in Figure 5, technology for reading information using barcodes or two-dimensional codes 2 (e.g., QR Code (registered trademark) or data matrix) is known to be highly reliable. For this reason, if a two-dimensional code 2 is printed on a document 1 output from a cash register (not shown) of a tenant's point-of-sale (POS) system, the scanner 30 and the portable imaging terminal 40 may each be configured to be able to scan the two-dimensional code 2.

[0058] 5, a case will be described in which the portable imaging terminal 40 captures an image of the form 1 and recognizes the visual information using the OCR function of the information recognition unit 11, and a case in which the portable imaging terminal 40 scans the two-dimensional code 2 written on the form 1. The two-dimensional code 2 written on the form 1 contains information that conforms to the standards and formats adopted by the owner. As shown in step #61 in FIG. 5, the tenant operator or the like first selects whether to capture an image of the form 1 or to scan the two-dimensional code 2 written on the form 1. In FIG. 5, the display of the portable imaging terminal 40 shows an imaging selection button 3 for selecting to capture an image of the form 1 and a scan selection button 4 for selecting to scan the two-dimensional code 2.

[0059] If the operator or the like touches the imaging selection button 3 in step #61, the operator or the like captures an image of the form 1 in step #62 (same as steps #05 to #09 in FIG. 2). Then, as shown in step #63 in FIG. 5, the tenant's operator or the like checks the visual information using the portable imaging terminal 40 and corrects any misreadings or omissions (same as steps #21 to #27 in FIG. 3). Furthermore, as shown in step #64 in FIG. 5, the owner's operator or the like checks the visual information using the confirmation terminal 50 and corrects any misreadings or omissions (same as steps #41 to #49 in FIG. 4).

[0060] If the operator or the like touches the scan selection button 4 in step #61, the operator or the like displays the form 1 on the display of the portable imaging terminal 40 and scans the two-dimensional code 2 in step #65. This alone enables the portable imaging terminal 40 to read the information on the form 1 from the two-dimensional code 2. The portable imaging terminal 40 transmits the form information read from the two-dimensional code 2 to the visual information recognition server 10. The visual information recognition server 10 then stores the information in the memory unit 12. Note that the portable imaging terminal 40 may be configured to transmit the form information read from the two-dimensional code 2 to the owner's sales management server.

[0061] In other words, if the operator or the like selects to scan the two-dimensional code 2 in step #61, there is no need for manual confirmation or correction work as shown in steps #63 and #64. This makes it even easier for tenants to report daily sales to the owner.

[0062] [Effects of the embodiment] The visual information recognition system of the present disclosure includes an imaging device (portable imaging terminal 40) that captures an image of a medium displaying visual information and generates image data of the medium, and a visual information processing device (visual information recognition server 10). The imaging device (portable imaging terminal 40) is configured to generate characteristic information that indicates the characteristics of the imaging device when capturing an image of the medium, and transmit the image data and characteristic information to the visual information processing device (visual information recognition server 10). The visual information processing device (visual information recognition server 10) includes an information recognition unit 11 that recognizes visual information from image data, and a storage unit 12 that stores the visual information and the characteristic information in association with each other.

[0063] With the above configuration, when the imaging device captures an image of a medium, characteristic information of the imaging device at the time the image was captured is also transmitted to the visual information processing device. The characteristic information can include not only the model and model number of the imaging device, but also version information of the operating system and applications, program behavior, etc. By storing such characteristic information in association with the visual information, it becomes possible to analyze the correlation between the quality of image data and the characteristic information. This makes it possible to provide a visual information recognition system that contributes to improving the quality of image data.

[0064] The visual information recognition system of the present disclosure is provided with reception units 41, 51 that receive corrections from an operator who confirms the accuracy of the visual information recognized by the information recognition unit 11. The storage unit 12 is configured to store correction information indicating the information on the corrections received by the reception units 41, 51 in association with the characteristic information.

[0065] With the above configuration, it is possible to understand the history of corrections based on the correction information. By associating the correction information with the characteristic information, it becomes easy to understand the correlation between the characteristic information of the imaging device and the superiority or inferiority of the recognition accuracy of visual information. This makes it easy to understand the compatibility between the imaging device and the visual information processing device.

[0066] In the visual information recognition system of the present disclosure, a visual information processing device (visual information recognition server 10) is provided with a performance information generation unit 13 that generates recognition performance information indicating the accuracy of visual information recognized from image data. A storage unit 12 is configured to store the recognition performance information in association with characteristic information.

[0067] With the above configuration, by associating the recognition performance information with the characteristic information, it becomes easier to understand the correlation between the characteristic information of the imaging device and the superiority or inferiority of the recognition accuracy of visual information, which makes it even easier to understand the compatibility between the imaging device and the visual information processing device.

[0068] In the visual information recognition system of the present disclosure, the storage unit 12 is configured to store the time when the imaging device captured an image of the medium in association with the recognition result information.

[0069] With the above configuration, it becomes easy to determine whether there is a difference in the recognition performance information between daytime and nighttime, for example.

[0070] In the visual information recognition system of the present disclosure, the medium is a form 1 on which the sales amount of a store is recorded.

[0071] With the above configuration, when acquiring visual information showing the sales amount of a store, the store operator or the like can know in advance which imaging device is compatible with the visual information processing device.

[0072] The visual information processing device (visual information recognition server 10) of the present disclosure is a visual information processing device (visual information recognition server 10) that acquires image data of a medium that displays visual information from an imaging device (portable imaging terminal 40) that images the medium, and characteristic information that indicates the characteristics of the imaging device when the imaging device (portable imaging terminal 40) images the medium, and is equipped with an information recognition unit 11 that recognizes visual information from the image data, and a memory unit 12 that stores the visual information and the characteristic information in association with each other.

[0073] With the above configuration, when the imaging device captures an image of a medium, characteristic information of the imaging device at the time the image was captured is also transmitted to the visual information processing device. The characteristic information can include not only the model and model number of the imaging device, but also version information of the operating system and applications, program behavior, etc. By storing such characteristic information in association with the visual information, it becomes possible to analyze the correlation between the quality of image data and the characteristic information. This makes it possible to provide a visual information processing device that contributes to improving the quality of image data.

[0074] The visual information processing method of the present disclosure includes an imaging step of imaging a medium displaying visual information with an imaging device (portable imaging terminal 40) and generating image data of the medium, a characteristic information generation step of generating characteristic information indicating the characteristics of the imaging device when the medium was imaged in the imaging step, an information recognition step of recognizing visual information from the image data, and a storage step of associating and storing the visual information and the characteristic information.

[0075] With the above configuration, when an imaging device captures an image of a medium, characteristic information of the imaging device at the time the image was captured is also generated. The characteristic information can include not only the model and model number of the imaging device, but also version information of the operating system and applications, program behavior, etc. By storing such characteristic information in association with visual information, it becomes possible to analyze the correlation between the quality of image data and the characteristic information. This realizes a visual information processing method that contributes to improving the quality of image data.

[0076] [Another embodiment] The present disclosure is not limited to the configurations exemplified in the above-described embodiments, and other representative embodiments of the present disclosure will be exemplified below.

[0077] (1) In the above-described embodiment, the medium is a document 1. However, the present invention is not limited to this embodiment, and the medium may be a gift certificate. The gift certificate may be imaged by a portable imaging terminal 40 or scanned by a scanner 30. The information recognition unit 11 recognizes visual information from the image data of the gift certificate and the image data of the gift certificate scanned by the scanner 30. The memory unit 12 stores the image data of the gift certificate, the image data of the gift certificate, and the visual information of the gift certificate recognized from the image data.

[0078] (2) In the above-described embodiment, a smartphone or a tablet terminal is exemplified as the imaging device (portable imaging terminal 40). However, the imaging device is not limited to this configuration, and may be, for example, a notebook computer with a camera module, a business-use PDA (personal digital assistant), or a feature phone.

[0079] (3) The visual information recognition server 10 may be configured integrally with, for example, a sales management server owned by the owner.

[0080] (4) In the above-described embodiment, the performance information generation unit 13 generates recognition performance information indicating the accuracy of visual information recognized from image data based on correction information received from the portable imaging terminal 40 or the confirmation terminal 50. The present invention is not limited to this embodiment, and the performance information generation unit 13 may be configured to generate recognition performance information based on the output results of a machine learning algorithm or artificial intelligence, for example.

[0081] (5) In the above-described embodiment, the visual information recognition server 10 is provided with the information recognition unit 11 and the storage unit 12. This is not limiting, and for example, the visual information processing device may be configured by a first server having the information recognition unit 11 and a second server having the storage unit 12. In addition, a third server having the score information generation unit 13 may also be included in the visual information processing device.

[0082] The configurations disclosed in the above-described embodiments (including other embodiments, the same applies hereinafter) can be applied in combination with configurations disclosed in other embodiments, as long as no contradiction occurs. Furthermore, the embodiments disclosed in this specification are merely examples, and the embodiments of the present disclosure are not limited thereto and can be appropriately modified within the scope of the present disclosure. [Industrial Applicability]

[0083] The present disclosure is applicable to a visual information recognition system, a visual information processing device, and a visual information processing method. [Explanation of symbols]

[0084] 1: Report 10: Visual information recognition server (visual information processing device) 11: Information recognition section 12: Storage section 13: Grade information generation section 40: Portable imaging terminal (imaging device) 41: Reception 51: Reception

Claims

1. an imaging device that captures an image of a medium that displays visual information and generates image data of the medium; a visual information processing device, the imaging device is configured to generate characteristic information indicating characteristics of the imaging device when the medium is imaged, and to transmit the image data and the characteristic information to the visual information processing device; The visual information processing device includes: an information recognition unit that recognizes the visual information from the image data; a storage unit that stores the visual information and the characteristic information in association with each other.

2. a reception unit that receives corrections from an operator to confirm whether the visual information recognized by the information recognition unit is correct, The visual information recognition system according to claim 1 , wherein the storage unit is configured to store correction information indicating information on the correction accepted by the accepting unit in association with the characteristic information.

3. the visual information processing device is provided with a performance information generation unit that generates recognition performance information indicating accuracy of the visual information recognized from the image data, The visual information recognition system according to claim 2 , wherein the storage unit is configured to store the recognition performance information in association with the characteristic information.

4. 4. The visual information recognition system according to claim 3, wherein the storage unit stores the time when the imaging device captured the image of the medium in association with the recognition result information.

5. 5. The visual information recognition system according to claim 1, wherein the medium is a form on which sales figures of a store are recorded.

6. The visual information recognition system according to claim 1 , wherein the medium is a gift certificate.

7. A visual information processing device that acquires, from an imaging device that has captured an image of a medium displaying visual information, image data of the medium and characteristic information that indicates characteristics of the imaging device when the imaging device captured the medium, an information recognition unit that recognizes the visual information from the image data; a storage unit that stores the visual information and the characteristic information in association with each other.

8. an imaging step of capturing an image of a medium displaying visual information with an imaging device and generating image data of the medium; a characteristic information generating step of generating characteristic information indicating characteristics of the imaging device when the medium is imaged in the imaging step; an information recognition step of recognizing the visual information from the image data; a storage step of storing the visual information and the characteristic information in association with each other.

Images