Visual information recognition device, visual information recognition system, visual information recognition method, and visual information recognition program

Abstract

To provide a visual information recognition device capable of recognizing all visual information displayed on a medium.SOLUTION: A visual information recognition device includes an imaging unit capable of imaging a medium 10 displaying visual information, an acquisition unit that acquires a moving image captured by the imaging unit, a visual information acquisition unit capable of acquiring visual information on the basis of acquisition target data that is at least one image data among a plurality of pieces of image data forming a moving image, and a cut-off determination unit that determines whether the medium 10 is cut off in the acquisition target data on the basis of the detection state of the edges 10U, 10D, 10L, and 10R of the medium 10 in the acquisition target data.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a visual information recognition device, a visual information recognition system, a visual information recognition method, and a visual information recognition program.

Background Art

[0002] For example, in many cases, the rent of a tenant occupying a retail store (so-called shopping mall, etc.) adopts a percentage system that varies according to the sales amount. In this case, since the landlord needs to accurately grasp the sales amount of the tenant, when receiving a sales report from the tenant, the landlord has been performing an operation to confirm the authenticity of the report. This confirmation operation requires a great deal of labor and time, and reducing the labor and time in the confirmation operation has been an issue. For example, in the information processing device disclosed in Patent Document 1, characters are imaged with a camera mounted on a mobile phone, and the characters are recognized using an OCR (Optical Character Recognition) function. In this information processing device, by moving the mobile phone along the text, the characters can be recognized one after another.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The configuration disclosed in Patent Document 1 allows, for example, a worker (such as a tenant's staff member) to capture images of a medium such as a sales slip containing a day's sales with a camera, thereby acquiring the visual information (text, etc.) on the medium as data and transmitting it to the landlord's management computer. In this respect, it can be useful as a means of solving the problem of reducing the labor and time required for the verification process mentioned above. However, if the sales slip submitted by the tenant is long, the worker needs to capture multiple photos or a continuous video with the camera from the beginning to the end of the sales slip. In this case, if the visual information displayed on the sales slip falls outside the camera's imaging range between the beginning and the end of the sales slip (if the text is cut off), there is a high possibility that the visual information will be misrecognized in the OCR process. Furthermore, if it cannot be confirmed that the sales slip has been properly captured from beginning to end, it could become a breeding ground for fraudulent activities such as underreporting of sales.

[0005] The object of the present invention is to provide a visual information recognition device that can recognize all visual information displayed on a medium without any omissions. [Means for solving the problem]

[0006] The visual information recognition device of the present invention for achieving the above objective is a visual information recognition device that continuously images a medium for displaying visual information, comprising: an imaging unit capable of imaging the medium; an acquisition unit that acquires a moving image captured by the imaging unit; and an acquisition target data which is at least one of a plurality of image data constituting the moving image, which is the visual information Our text information of OCR processing The system includes a visual information acquisition unit that can acquire visual information, and a cropping determination unit that determines whether or not cropping of the medium occurs in the acquired data based on the detection state of the edge of the medium in the acquired data, and the visual information acquisition unit Acquisition of the aforementioned character information The processing of the aforementioned part and the processing of the part-of-view determination unit are performed simultaneously.

[0007] According to the above configuration, the cropping detection unit determines whether or not there is cropping in the media captured in the moving image. This configuration reduces the risk of misrecognizing visual information due to cropping when the visual information acquisition unit acquires visual information. Furthermore, with this configuration, by detecting the edges of the media, it is possible to confirm that the media has been properly imaged from beginning to end. As a result, a visual information recognition device that can recognize all visual information displayed on the media is realized.

[0008] The present invention is not limited to visual information recognition devices, but can also be applied to visual information recognition systems. To achieve the above objective, the visual information recognition system of the present invention is a visual information recognition system that acquires moving images by continuously capturing images of a medium that displays visual information, and comprises an acquisition unit that acquires the moving images captured of the medium, and an acquisition target data which is at least one of a plurality of image data constituting the moving image, and visual information Our text information of OCR processing The system includes a visual information acquisition unit that can acquire visual information, and a cropping determination unit that determines whether or not cropping of the medium occurs in the acquired data based on the detection state of the edge of the medium in the acquired data, and the visual information acquisition unit Acquisition of the aforementioned character information The processing of the aforementioned part and the processing of the part-of-view determination unit are performed simultaneously.

[0009] The present invention is also applicable to methods for recognizing visual information. To achieve the above objective, the present invention provides a method for recognizing visual information in a visual information recognition system that acquires moving images obtained by continuously capturing images of a medium that displays visual information, comprising: an acquisition step of acquiring the moving images captured of the medium; and acquiring target data which is at least one of a plurality of image data constituting the moving image, which is the visual information Our text information of OCR processing The visual information acquisition step includes a visual information acquisition step and a cropping determination step that determines whether or not cropping of the medium occurs in the acquired data based on the detection state of the edge of the medium in the acquired data, and the visual information acquisition step Acquisition of the aforementioned character information and the aforementioned cutoff determination step Determination of whether or not the aforementioned medium is cut off. It is characterized by the fact that these processes are carried out simultaneously.

[0010] The present invention is also applicable to visual information recognition programs. To achieve the above objective, the present invention is a visual information recognition program that acquires moving images obtained by continuously capturing images of a medium that displays visual information, and includes an acquisition function for acquiring the moving images captured of the medium, and an acquisition target data which is at least one of a plurality of image data that constitutes the moving image, which is the visual information Our text information of OCR processing The computer is made to perform a visual information acquisition function and a cropping determination function that determines whether or not cropping of the medium occurs in the target data based on the detection state of the edge of the medium in the target data to be acquired, and the visual information acquisition function Acquisition of the aforementioned character information and the aforementioned cutoff detection function Determination of whether or not the aforementioned medium is cut off. It is characterized by being configured to run concurrently.

[0011] Another characteristic configuration of the visual information recognition device according to the present invention is that the visual information acquisition unit is configured to acquire the visual information that is located within a preset identification range in the imaging range of the data to be acquired, and the cropping determination unit is configured to determine whether or not cropping has occurred by detecting or not the edge within the identification range.

[0012] With the above feature configuration, the identification range is predetermined within the imaging range of the data to be acquired, which is preferable because it allows for the proper acquisition of visual information and detection of edges.

[0013] Another characteristic configuration of the visual information recognition device according to the present invention is that it is equipped with a notification unit capable of notifying the operator, the cropping determination unit is configured to determine a first cropping state in which cropping occurs when the upper edge of the medium is not detected, and when the cropping determination unit determines the cropping state, it is configured to prohibit the visual information acquisition unit from starting to acquire the visual information, and to instruct the notification unit to notify that cropping has occurred.

[0014] If the upper edge of the media is not detected, the visual information acquisition unit may not include characters or other elements near the upper edge of the media in the target data, potentially resulting in improper acquisition of visual information. With the above feature configuration, if the upper edge of the media is not detected, the start of visual information acquisition is prohibited, preventing the acquisition of visual information under inappropriate conditions. Alternatively, a notification is issued to encourage the operator to take appropriate images. Therefore, it is possible to achieve proper acquisition of visual information from near the upper edge of the media.

[0015] Another characteristic configuration of the visual information recognition device according to the present invention is that it is equipped with a notification unit capable of notifying the operator, the cropping determination unit is configured to determine a second cropping state in which cropping occurs when it does not detect at least one of the left and right edges of the medium, and when the cropping determination unit determines the second cropping state, it is configured to prohibit the visual information acquisition unit from starting to acquire the visual information, and to instruct the notification unit to notify that cropping has occurred.

[0016] If the left and right edges of the media are not detected, text and other elements near those edges may not be included in the acquired data, potentially leading to improper acquisition of visual information. With the above feature configuration, if the left and right edges of the media are not detected, the start of visual information acquisition is prohibited, preventing the acquisition of visual information under inappropriate conditions. Alternatively, a notification is issued to encourage the operator to take appropriate photos. Therefore, it is possible to achieve proper acquisition of visual information from the vicinity of the left and right edges of the media.

[0017] Another characteristic configuration of the visual information recognition device according to the present invention is that it is equipped with an operation unit that accepts human operation by the operator, the cropping determination unit is configured to determine a third cropping state in which cropping occurs when the lower edge of the medium is not detected, and when the cropping determination unit determines the third cropping state, it is configured to give at least one of the following instructions: prohibit the visual information acquisition unit from terminating the acquisition of the visual information, and instruct the notification unit to notify that cropping has occurred when the operation unit is subjected to human operation.

[0018] For example, when acquiring a moving image by capturing from the top edge to the bottom edge of a medium with the imaging unit, it is possible that the imaging may end before the bottom edge is fully captured, resulting in cropping at the bottom edge. In this case, text or other elements near the bottom edge of the medium may not be included in the acquired data, and the acquisition of visual information may not be performed properly. With the above feature configuration, if the bottom edge of the medium is not detected, the termination of visual information acquisition is prohibited, preventing the termination of visual information acquisition in an inappropriate state. Alternatively, a notification is issued to prompt the operator to take appropriate images. Therefore, it is possible to achieve proper acquisition of visual information from near the bottom edge of the medium. Note that the cropping detection unit may be configured to issue a notification when the operating unit is manually operated, or it may be configured to issue a notification when the cropping detection unit determines a third cropping state after the operating unit has already been operated.

[0019] Another characteristic configuration of the visual information recognition device according to the present invention is that the cropping determination unit is configured to determine a fourth cropping state in which cropping has occurred when it stops detecting at least one of the left and right edges of the medium after the visual information acquisition unit has started acquiring the visual information, and when the cropping determination unit determines the fourth cropping state, it is configured to prohibit the visual information acquisition unit from terminating the acquisition of the visual information, and to instruct the notification unit to notify that cropping has occurred, until the cropping determination unit no longer determines that the fourth cropping state has occurred.

[0020] For example, when the medium is a sales invoice that is long vertically, it is conceivable that the operator moves the imaging unit vertically while imaging the medium over its entire vertical extent. With the above-described characteristic configuration, even if a break occurs at the left and right edges of the medium during imaging of the medium, acquisition of visual information by the visual information acquisition unit cannot be terminated until the break is resolved. As a result, the medium shown in the moving image is in a state where its left and right edges are shown over its entire vertical extent, further reducing the risk that the visual information acquisition unit misrecognizes the visual information.

[0021] Another characteristic configuration of the visual information recognition apparatus according to the present invention is that, after the visual information acquisition unit starts acquiring the visual information, when the break determination unit stops detecting at least one of the left and right edges of the medium, the break determination unit is configured to determine a fourth break state in which a break has occurred. When the break determination unit determines the fourth break state, the break determination unit is configured to perform at least one of canceling acquisition of the visual information by the visual information acquisition unit and instructing the notification unit to notify that a break has occurred.

[0022] With the above-described characteristic configuration, when a break occurs at the left and right edges of the medium during imaging of the medium, it becomes possible to prompt the operator to re-image the medium. This makes it easier for the operator to appropriately image the medium.

[0023] Another characteristic configuration of the visual information recognition apparatus according to the present invention is that, after the visual information acquisition unit starts acquiring the visual information, when the break determination unit stops detecting the medium at all, the break determination unit is configured to determine a fifth break state in which a break has occurred. When the break determination unit determines the fifth break state, the break determination unit is configured to perform at least one of canceling acquisition of the visual information by the visual information acquisition unit and instructing the notification unit to notify that a break has occurred.

[0024] With the above-described characteristic configuration, when the medium completely disappears during imaging of the medium, it becomes possible to prompt the operator to re-image the medium. This makes it easier for the operator to appropriately image the medium. [Brief explanation of the drawing]

[0025] [Figure 1] This is a block diagram showing an example of a visual information recognition device. [Figure 2] This is a flowchart illustrating an example of processing in a visual information recognition device. [Figure 3] This diagram shows the media and the identification range. [Figure 4] This diagram shows a situation where the right edge of the media is cropped. [Figure 5] This diagram shows a situation where the left edge of the media is cropped. [Modes for carrying out the invention]

[0026] [Configuration of the visual information recognition device] As shown in Figure 1, the visual information recognition device of this embodiment is equipped with a camera 1, an operation unit 2, a video input unit 3, an out-of-view detection unit 4, a visual information acquisition unit 5, a notification unit 6, and a display unit 7. This visual information recognition device may be configured to be incorporated into, for example, a portable information terminal such as a smartphone, mobile computer, or tablet computer, or a stationary device (desktop PC or POS device).

[0027] Camera 1 is configured to capture images from a medium 10, such as a receipt or sales slip. Visual information (characters, symbols, and barcodes, etc.) is displayed on the medium 10. The operation unit 2 accepts human input from the operator to the visual information recognition device. The video input unit 3 inputs the moving image captured by camera 1 as data. Camera 1 corresponds to the "imaging unit" of the present invention. The video input unit 3 corresponds to the "acquisition unit" of the present invention.

[0028] The video is composed of multiple image data (image data for each frame). The crop detection unit 4 is configured to extract the medium 10 from the multiple image data. The image data in which the medium 10 is captured becomes the data to be acquired by the visual information acquisition unit 5 for acquiring visual information. The crop detection unit 4 is also configured to detect the edges of the medium 10. The edges of the medium 10 are the upper edge 10U, lower edge 10D, left edge 10L, and right edge 10R, as shown in Figure 3. Examples of methods for detecting the edges of the medium 10 include image analysis processing that distinguishes the medium 10 from the background by color and brightness, and processing that detects marks placed near the edges of the medium 10.

[0029] When the cropping detection unit 4 detects the edges of the medium 10, or when the visual information acquisition unit 5 (described later) acquires visual information, these processes may be performed simultaneously with the imaging by the camera 1. In this case, real-time processing is required for the processing by the cropping detection unit 4 and the visual information acquisition unit 5. For this reason, in this embodiment, a detection range of the medium 10 is set to an identification range C (see Figures 3 to 5) that is narrower than the imaging range of the camera 1. This makes it easier to ensure real-time processing by the cropping detection unit 4 and the visual information acquisition unit 5 compared to a configuration where the identification range C is the entire imaging range of the camera 1.

[0030] The cropping determination unit 4 determines whether cropping of the medium 10 has occurred in the data to be acquired, based on the detection status of the edge of the medium 10 in the data to be acquired. The detection status of the edge of the medium 10 is whether or not the visual information acquisition unit 5 detects the edge within the identification range C. Therefore, cropping of the medium 10 in this embodiment means that the edge of the medium 10 is outside the identification range C, as shown in Figures 4 and 5, for example. In other words, the cropping determination unit 4 is configured to determine whether or not cropping has occurred by detecting whether or not the edge is within the range of the identification range C. In this embodiment, the state in which the edge of the medium 10 is outside the identification range C is referred to as the "cropped state".

[0031] When the cropping detection unit 4 determines that the media 10 is cropped, the notification unit 6 is configured to notify the operator that cropping has occurred. The notification unit 6 may be, for example, an audio device including a buzzer or speaker, a display device including a display panel or lamp, or a device that includes both an audio device and a display device.

[0032] The visual information acquisition unit 5 is configured to acquire visual information that exists within a preset identification range C within the imaging range of the data to be acquired. In other words, the visual information acquisition unit 5 acquires visual information based on the data to be acquired from among multiple image data that constitute the moving image. For the acquisition of text information in particular, the so-called well-known OCR (Optical Character Recognition) function and the AI-OCR function that utilizes artificial intelligence are used. For example, the OCR function is used for the text information acquisition process, and the AI-OCR function is used to perform a comparison process to determine whether or not there are misrecognitions in the acquired text information.

[0033] The display unit 7 is configured to display at least one of the acquired video, the image data constituting the video, and the data to be acquired. The display unit 7 may also be capable of displaying the identification range C. The display unit 7 is, for example, a liquid crystal display or OLED display provided in the aforementioned portable information terminal or stationary device. The touch panel display provided in the aforementioned portable information terminal or stationary device may be configured to serve as both the display unit 7 and the operation unit 2, the display unit 7 and the notification unit 6, or the display unit 7, the operation unit 2, and the notification unit 6.

[0034] [Flow of acquiring visual information displayed on a medium] Based on Figures 2 to 5, the acquisition of visual information displayed on the medium 10 and the determination of the state in which the medium 10 is partially obscured during the acquisition of visual information will be explained. As shown in Figure 3, for example, the operator continuously captures images of the upper and lower ends of the medium 10 with the camera 1. At this time, the visual information recognition device of this embodiment executes the process shown in the flowchart of Figure 2.

[0035] When the operator starts imaging with camera 1 (step #01), the video input unit 3 acquires at least one of the multiple image data that make up the image captured by camera 1, i.e., the moving image (step #02). Once the moving image is acquired by the video input unit 3, the cropping detection unit 4 detects whether or not the upper edge 10U of the medium 10 exists in the identification range C. That is, the cropping detection unit 4 determines whether or not it has detected the upper edge 10U of the medium 10 (step #03-1).

[0036] If the cropping detection unit 4 does not detect the upper edge 10U of the medium 10 (Step #03-1: No), the cropping detection unit 4 determines the cropping state of the medium 10. This cropping state is referred to as the "first cropping state." In other words, the cropping detection unit 4 is configured to determine the first cropping state, where cropping occurs, when it does not detect the upper edge 10U of the medium 10. The cropping detection unit 4 then sends a notification instruction signal to the notification unit 6, and the notification unit 6 notifies that cropping has occurred (Step #04). At this time, the visual information acquisition unit 5 does not start acquiring visual information and returns to the process in Step #02. In other words, when the cropping detection unit 4 determines the first cropping state, it is configured to prohibit the visual information acquisition unit 5 from starting to acquire visual information and to instruct the notification unit 6 to notify that cropping has occurred.

[0037] If the cropping detection unit 4 has detected the upper edge 10U of the medium 10 (Step #03-1: Yes), the cropping detection unit 4 detects whether both the left edge 10L and the right edge 10R of the medium 10 are present in the identification range C of the multiple image data constituting the moving image. That is, the cropping detection unit 4 determines whether both the left edge 10L and the right edge 10R of the medium 10 have been detected (Step #03-2).

[0038] If the cropping detection unit 4 does not detect at least one of the left edge 10L and the right edge 10R of the medium 10 (Step #03-2: No), the cropping detection unit 4 determines the cropping state of the medium 10. This cropping state is referred to as the "second cropping state." In other words, the cropping detection unit 4 is configured to determine the second cropping state, in which cropping occurs, when it does not detect at least one of the left edge 10L and the right edge 10R (left and right edges) of the medium 10. The cropping detection unit 4 then sends a notification instruction signal to the notification unit 6, and the notification unit 6 notifies that cropping has occurred (Step #04). In other words, when the cropping detection unit 4 determines the second cropping state, it is configured to prohibit the start of visual information acquisition by the visual information acquisition unit 5 and to instruct the notification unit 6 to notify that cropping has occurred.

[0039] If the cropping detection unit 4 detects both the left edge 10L and the right edge 10R of the medium 10 (Step #03-2: Yes), the visual information acquisition unit 5 starts acquiring visual information (Step #05). Once the visual information acquisition unit 5 starts acquiring visual information, the video input unit 3 acquires at least one of the multiple image data that make up the captured image from the camera 1, i.e., the moving image, similar to the process in Step #02 (Step #06).

[0040] When a video input unit 3 acquires a video image, the cropping detection unit 4 detects whether both the left edge 10L and the right edge 10R of the medium 10 are present in the identification range C of the multiple image data constituting the video image. That is, the cropping detection unit 4 determines whether both the left edge 10L and the right edge 10R of the medium 10 have been detected (step #07).

[0041] If the cropping detection unit 4 does not detect at least one of the left edge 10L and the right edge 10R of the medium 10 (Step #07: No), the cropping detection unit 4 determines the cropping state of the medium 10. This cropping state is referred to as the "fourth cropping state." In other words, the cropping detection unit 4 is configured to determine the fourth cropping state, in which cropping has occurred, when it stops detecting at least one of the left edge 10L and the right edge 10R of the medium 10 after the visual information acquisition unit 5 has started acquiring visual information. The cropping detection unit 4 then sends a notification instruction signal to the notification unit 6, and the notification unit 6 notifies that cropping has occurred (Step #08). At this time, the cropping detection unit 4 sends a notification signal to the visual information acquisition unit 5 to prohibit the acquisition of visual information, and the visual information acquisition unit 5 temporarily stops acquiring visual information.

[0042] When the cropping detection unit 4 determines the fourth cropping state (step #07: No) and the notification unit 6 makes a notification (step #08), the cropping detection unit 4 detects whether or not the medium 10 exists in the identification range C of the multiple image data constituting the moving image. That is, it determines whether or not the cropping detection unit 4 has stopped detecting the medium 10 at all (step #09). If the cropping detection unit 4 has detected even a part of the medium 10 in step #09 (step #09: No), it returns to the process in step #06. In other words, even if the cropping detection unit 4 does not detect at least one of the left edge 10L and the right edge 10R of the medium 10, if it has detected even a part of the medium 10, the process loops in the order of steps #07, step #08, step #09, and step #06.

[0043] For example, as shown in Figure 4, if the right edge 10R is cut off from the identification range C, the visual information acquisition unit 5 may misinterpret the displayed price (visual information), such as "¥155,800," as "¥155." Also, as shown in Figure 5, if the left edge 10L is cut off from the identification range C, the visual information acquisition unit 5 may misinterpret the displayed text (visual information), such as "average unit price," as "unit price." For this reason, when the cut-off determination unit 4 determines that the fourth cut-off state has occurred, it is configured to prohibit the acquisition of visual information by the visual information acquisition unit 5 and to instruct the notification unit 6 to notify that a cut-off has occurred, until the cut-off determination unit 4 no longer determines that the fourth cut-off state has occurred.

[0044] At this time, the operator images the medium 10 by moving the camera 1 left and right so that both the left edge 10L and the right edge 10R are included in the identification range C. When the cropping detection unit 4 detects both the left edge 10L and the right edge 10R of the medium 10 (Step #07: Yes), the visual information acquisition unit 5 acquires visual information (Step #10).

[0045] When the cropping detection unit 4 no longer detects the medium 10 at all (Step #09: Yes), the cropping detection unit 4 determines the cropping state of the medium 10. This cropping state is referred to as the "fifth cropping state." In other words, the cropping detection unit 4 is configured to determine the fifth cropping state, which is the state in which cropping occurs, when the visual information acquisition unit 5 has started acquiring visual information and no longer detects the medium 10 at all.

[0046] When the cropping detection unit 4 determines that the fifth cropping state has occurred, it sends an instruction signal to the visual information acquisition unit 5 to cancel the acquisition of visual information, and the visual information acquisition unit 5 cancels the acquisition of visual information (step #15). At this time, the visual information acquisition unit 5 deletes the data related to the visual information acquired from step #05 onwards. Also in step #15, the cropping detection unit 4 sends an instruction signal to the notification unit 6 to give a notification. The notification unit 6 then gives a notification that the entire medium 10 is cropped and prompts the operator to start again from step #01. In other words, when the cropping detection unit 4 determines that the fifth cropping state has occurred, it is configured to instruct the visual information acquisition unit 5 to cancel the acquisition of visual information and the notification unit 6 to give a notification that cropping has occurred. Note that although imaging by camera 1 ends in step #15, the camera 1 may continue to take images. If the camera 1 continues to take images, the system may return to step #02 as described above.

[0047] After a Yes determination is made in step #07 and step #10 is processed, the cropping determination unit 4 determines whether or not it has detected human operation of the operation unit 2 (step #11). The operation unit 2 is, for example, a push-button switch or a software switch displayed on a touch panel operation monitor, which accepts human operation to terminate the acquisition of visual information displayed on the medium 10. If no human operation of the operation unit 2 is detected (step #11: No), the process returns to step #06.

[0048] While the operator moves camera 1 and images are taken across the upper edge 10U and lower edge 10D of the medium 10, the processes described in steps #06 to #11 above are looped. If a "Yes" determination is made in step #09 during this process, the process of acquiring the visual information displayed on the medium 10 is stopped.

[0049] If human operation of the operation unit 2 is detected (Step #11: Yes), the cropping determination unit 4 detects whether or not the lower edge 10D of the medium 10 exists in the identification range C. That is, it determines whether or not the cropping determination unit 4 has detected the lower edge 10D of the medium 10 (Step #12).

[0050] If the cropping detection unit 4 does not detect the lower edge 10D of the medium 10 (Step #12: No), the cropping detection unit 4 determines the cropping state of the medium 10. This cropping state is referred to as the "third cropping state." In other words, the cropping detection unit 4 is configured to determine the third cropping state, where cropping occurs, when it does not detect the lower edge 10D of the medium 10. The cropping detection unit 4 then sends a notification instruction signal to the notification unit 6, and the notification unit 6 notifies that cropping has occurred (Step #13). Furthermore, the acquisition of visual information by the visual information acquisition unit 5 does not end, and the process returns to Step #06, and the processing of Steps #06 to #11 described above is looped. In other words, when the cropping detection unit 4 determines the third cropping state, it is configured to prohibit the completion of the acquisition of visual information by the visual information acquisition unit 5, and to instruct the notification unit 6 to notify that cropping has occurred due to human operation of the operation unit 2.

[0051] While the processes described in steps #06 to #11 loop, the cropping detection unit 4 is configured to determine a fourth cropping state when it stops detecting at least one of the left edge 10L and the right edge 10R of the medium 10 (step #07: No). Therefore, when the cropping detection unit 4 determines a fourth cropping state, it is configured to prohibit the visual information acquisition unit 5 from terminating the acquisition of visual information and to instruct the notification unit 6 to notify that cropping has occurred, until the cropping detection unit 4 no longer determines that a fourth cropping state has occurred.

[0052] When the cropping detection unit 4 detects the lower edge 10D of the medium 10 (Step #12: Yes), the visual information acquisition process by the visual information acquisition unit 5 ends (Step #14). At this time, imaging by camera 1 ends, but the configuration may continue imaging by camera 1. If the configuration continues imaging by camera 1, the system may return to Step #02 described above.

[0053] [Another embodiment] The present invention is not limited to the configurations exemplified in the embodiments described above, and other representative embodiments of the present invention are described below.

[0054] (1) In the above embodiment, when the cropping determination unit 4 determines the first cropping state, it is configured to prohibit the start of visual information acquisition by the visual information acquisition unit 5 and to instruct the notification unit 6 to notify that cropping has occurred. The embodiment is not limited to this one, and for example, when the cropping determination unit 4 determines the first cropping state, it may be configured to prohibit the start of visual information acquisition by the visual information acquisition unit 5 and to instruct the notification unit 6 to notify that cropping has occurred.

[0055] (2) In the above embodiment, when the cropping determination unit 4 determines the second cropping state, it is configured to prohibit the start of visual information acquisition by the visual information acquisition unit 5 and to instruct the notification unit 6 to notify that cropping has occurred. The embodiment is not limited to this one, and for example, when the cropping determination unit 4 determines the second cropping state, it may be configured to prohibit the start of visual information acquisition by the visual information acquisition unit 5 and to instruct the notification unit 6 to notify that cropping has occurred.

[0056] (3) In the above embodiment, when the cropping determination unit 4 determines the third cropping state, it is configured to prohibit the visual information acquisition unit 5 from terminating the acquisition of visual information and to instruct the notification unit 6 to notify that cropping has occurred when the operation unit 2 is subjected to human operation. The embodiment is not limited to this one, and for example, when the cropping determination unit 4 determines the third cropping state, it may be configured to prohibit the visual information acquisition unit 5 from terminating the acquisition of visual information and to instruct the notification unit 6 to notify that cropping has occurred when the operation unit 2 is subjected to human operation.

[0057] (4) In the above embodiment, when the cropping determination unit 4 determines that the fourth cropping state has occurred, it is configured to prohibit the acquisition of visual information by the visual information acquisition unit 5 and to instruct the notification unit 6 to notify that cropping has occurred until it is no longer determined that the fourth cropping state has occurred. The embodiment is not limited to this one, and for example, when the cropping determination unit 4 determines that the fourth cropping state has occurred, it may be configured to prohibit the acquisition of visual information by the visual information acquisition unit 5 and to instruct the notification unit 6 to notify that cropping has occurred until it is no longer determined that the fourth cropping state has occurred. Alternatively, for example, when the cropping determination unit 4 determines that the fourth cropping state has occurred, it may be configured to cancel the acquisition of visual information by the visual information acquisition unit 5 and to instruct the notification unit 6 to notify that cropping has occurred. If the acquisition of visual information by the visual information acquisition unit 5 is canceled, the visual information acquisition unit 5 deletes the data related to the visual information acquired from step #05 onwards in Figure 2. In this case, the notification unit 6 may also be configured to provide notification to the operator prompting them to restart from step #01.

[0058] (5) In the above embodiment, when the cropping determination unit 4 determines the fifth cropping state, it is configured to cancel the acquisition of visual information by the visual information acquisition unit 5 and to instruct the notification unit 6 to notify that cropping has occurred. The embodiment is not limited to this one, and for example, when the cropping determination unit 4 determines the fifth cropping state, it may be configured to cancel the acquisition of visual information by the visual information acquisition unit 5 and to instruct the notification unit 6 to notify that cropping has occurred.

[0059] (6) In the above-described embodiment, as shown in steps #11 and #12 of Figure 2, when human operation of the operation unit 2 is detected, the cut-off determination unit 4 determines whether or not the lower edge 10D of the medium 10 has been detected. The embodiment is not limited to this, and for example, the operation unit 2 does not have to be configured to accept human operation for the termination of the acquisition of visual information displayed on the medium 10. Also, for example, the configuration may not include an operation unit 2. For example, after the processing in step #10 of Figure 2, the determination in step #12 may be performed without going through step #11. And, when the cut-off determination unit 4 detects the lower edge 10D of the medium 10 (step #12: Yes), the configuration may be such that the visual information acquisition process by the visual information acquisition unit 5 is terminated (step #14).

[0060] (7) In the above-described visual information recognition device, an example configuration was given in which the camera 1, the operation unit 2, the video input unit 3, the cropping detection unit 4, the visual information acquisition unit 5, and the notification unit 6 are incorporated into a smartphone or mobile computer, but the embodiment is not limited to this. For example, the visual information acquisition unit 5 may be built on a dedicated server, and video footage of the medium 10 captured by a mobile terminal such as a smartphone or mobile computer is transmitted to the dedicated server as data, and the visual information acquisition unit 5 on the dedicated server acquires visual information from the video footage in a visual information recognition system. In such a visual information recognition system, when the mobile terminal captures the medium 10, only cropping detection of the edges of the medium 10 is performed by the cropping detection unit 4, and after the capture of the medium 10 is completed, video footage of the medium 10 with the top, bottom, left, and right edges captured is transmitted to the dedicated server as data. Furthermore, as a result of the visual information acquisition process by the visual information acquisition unit 5, various data acquired based on the visual information acquisition process is transmitted to the mobile terminal as data and displayed on the screen of the mobile terminal.

[0061] If the video footage from the medium 10 is transmitted to a dedicated server, the amount of data transmitted will increase, impairing the real-time performance of the visual information acquisition process by the visual information acquisition unit 5. To resolve this problem, while the operator is capturing images from the medium 10, the processing unit of the mobile terminal may extract only the portion of the video footage that displays visual information from each frame of the video footage, and transmit the extracted portion of the visual information to the dedicated server. Since the extracted portion of the visual information is smaller than the capture range of the video footage, the amount of data transmitted is reduced, improving the real-time performance of the visual information acquisition process by the visual information acquisition unit 5.

[0062] (8) In the above-described embodiment, the visual information acquisition process by the visual information acquisition unit 5 is performed by using frame-by-frame image data from the video captured by the camera 1, but the embodiment is not limited to this. For example, the visual information acquisition process by the visual information acquisition unit 5 may be configured to be performed by using a single panoramic image data showing the entire medium 10. For example, the image data from each frame of the video captured by the camera 1 may be matched with the edges of the medium 10 and visual information, and a single panoramic image data showing the entire medium 10 may be generated. This makes it easy to visually confirm whether the medium 10 has been properly captured during or after the camera 1 captures the medium 10, and allows the operator to easily confirm which part of the medium 10 was not properly captured. Furthermore, if video is saved as evidence that the medium 10 has been captured, there is a risk that the storage capacity will increase, but if a single panoramic image data showing the entire medium 10 is saved as evidence, the evidence that the medium 10 has been captured can be saved with the minimum necessary storage capacity.

[0063] (9) The present invention is also applicable to the visual information recognition method in the visual information recognition device and visual information recognition system described above. In this case, steps #02 and #06 described above correspond to the "acquisition step". Steps #03-1, #03-2, #07, #09 and #12 described above correspond to the "partial view determination step". Steps #05 and #10 described above correspond to the "visual information acquisition step".

[0064] (10) The present invention is also applicable to a visual information recognition program to be executed by a computer in the above-described visual information recognition device or visual information recognition system. In this case, the above-described video input unit 3 has an "acquisition function". The above-described cropped-out determination unit 4 has a "cropped-out determination function". The above-described visual information acquisition unit 5 has a "visual information acquisition function".

[0065] (11) In the embodiments described above, receipts and sales slips from transactions were given as examples of media 10, but media 10 may also be, for example, medical questionnaires or health checkup reports.

[0066] (12) In the above-described embodiment, an identification range C (see Figures 3 to 5) that is narrower than the imaging range of the camera 1 is pre-set as the detection range of the medium 10. The embodiment is not limited to this, and the identification range C may extend over the entire imaging range of the camera 1.

[0067] (13) In the embodiment described above based on Figure 2, the obscuration determination unit 4 instructs the notification unit 6 to notify when the operation unit 2 is operated manually, but the embodiment is not limited to this. For example, the configuration may be such that the obscuration determination unit 4 determines the third obscuration state after the operation unit 2 has already been operated, and then instructs the notification unit 6 to notify.

[0068] (14) In the embodiment described above based on Figure 3, the operator scrolls the camera 1 from top to bottom to capture images across the upper edge 10U and lower edge 10D of the medium 10, but the embodiment is not limited to this. The operator may also scroll the camera 1 from bottom to top to capture images across the upper edge 10U and lower edge 10D of the medium 10. In this case, the first cropped state is the state in which the lower edge 10D of the medium 10 is not detected. In this case, the third cropped state is the state in which the upper edge 10U of the medium 10 is not detected.

[0069] For example, if the medium 10 has a horizontal configuration (the left edge 10L and right edge 10R are longer than the upper edge 10U and lower edge 10D), the operator may scroll the camera 1 left and right to capture images across the left edge 10L and right edge 10R of the medium 10. In this case, the first cropping state is when one of the left edge 10L and right edge 10R of the medium 10 is not detected. In this case, the second cropping state is when at least one of the upper edge 10U and lower edge 10D of the medium 10 is not detected. In this case, the third cropping state is when the other of the left edge 10L and right edge 10R of the medium 10 is not detected. In this case, the fourth cropping state is when, after the visual information acquisition unit 5 has started acquiring visual information, at least one of the upper edge 10U and lower edge 10D of the medium 10 is no longer detected.

[0070] [Variation] In the above-described embodiment, the cropping determination unit 4 determines whether cropping of the medium 10 has occurred based on the detection state of the edges of the medium 10 in the data to be acquired. The cropping determination unit 4 may be configured to detect, for example, the area in which a string of characters exists on the medium 10. The shape of the area in which the string of characters exists may be a rectangle or any other shape. The cropping determination unit 4 may also be configured to determine whether the left and right edges of the area in which the string of characters exists are located in the left and right regions of the imaging range of the camera 1. In this case, the cropping determination unit 4 determines that cropping has occurred on the medium 10 if at least one of the left and right edges of the area in which the string of characters exists is located in at least one of the left and right edges of the imaging range of the camera 1. For example, a threshold range for cropping determination is set in the left-right direction from the left and right edges of the imaging range of the camera 1, within a range of a predetermined number of pixels. The cropping determination unit 4 determines cropping if the area in which the string of characters exists overlaps with the threshold range. The cropping determination unit 4 also determines that cropping has not occurred if the area in which the string of characters exists does not overlap with the threshold range. In other words, the cropping determination unit 4 may be configured to determine whether or not cropping of the medium 10 has occurred based on the detection status of the medium 10 in the data to be acquired.

[0071] Furthermore, the configurations disclosed in the above-described embodiments (including other embodiments, the same applies hereinafter) can be applied in combination with configurations disclosed in other embodiments, as long as no inconsistencies arise. In addition, the embodiments disclosed herein are illustrative, and the embodiments of the present invention are not limited thereto, and can be modified as appropriate without departing from the object of the present invention. [Industrial applicability]

[0072] The present invention is applicable to a visual information recognition device equipped with an imaging unit capable of capturing images of a medium for displaying visual information, a visual information recognition system composed of two or more devices, a visual information recognition method, and a visual information recognition program that is incorporated into the above-mentioned visual information recognition device or visual information recognition system, or that causes a computer to execute the above-mentioned visual information recognition method. [Explanation of Symbols]

[0073] 1: Camera (imaging unit) 2 :Operation section 3: Video input unit (acquisition unit) 4: Cut-off detection unit 5: Visual Information Acquisition Unit 6: Hochi Department 10: Medium 10U: Upper edge (edge) 10D: Lower edge (edge) 10L:Left edge (edge) 10R: Right edge (edge) C: Identification range

Claims

1. A visual information recognition device that continuously captures images of a medium that displays visual information, An imaging unit capable of imaging the aforementioned medium, An acquisition unit that acquires moving images captured by the aforementioned imaging unit, A visual information acquisition unit capable of acquiring character information from the visual information by OCR processing based on acquisition target data which is at least one of the multiple image data that constitute the moving image, The system includes a cropping determination unit that determines whether or not cropping of the medium occurs in the acquired data based on the detection state of the edge of the medium in the acquired data, A visual information recognition device in which the process of acquiring the character information by the visual information acquisition unit and the process of determining the cropped portion of the image are performed simultaneously in parallel.

2. The visual information acquisition unit is configured to acquire the visual information that is present within a preset identification range in the imaging range of the data to be acquired. The visual information recognition device according to claim 1, wherein the cropping determination unit is configured to determine whether or not cropping has occurred by detecting or not the edge within the identification range.

3. It is equipped with a notification unit capable of notifying the operator, The cropping detection unit is configured to determine a first cropping state in which cropping occurs when it does not detect the upper edge of the medium. The visual information recognition device according to claim 1 or 2, wherein the cropping determination unit is configured to, upon determining the cropping state, to prohibit the visual information acquisition unit from starting to acquire the visual information, and to instruct the notification unit to notify that cropping has occurred.

4. It is equipped with a notification unit capable of notifying the operator, The cropping detection unit is configured to determine a second cropping state in which cropping occurs when it does not detect at least one of the left or right edges of the medium. The visual information recognition device according to any one of claims 1 to 3, wherein the cropping determination unit is configured to, upon determining the second cropping state, to prohibit the visual information acquisition unit from starting to acquire the visual information, and to instruct the notification unit to notify that cropping has occurred.

5. The system is equipped with an operating unit that accepts manual input from the operator, The cropping detection unit is configured to determine a third cropping state in which cropping occurs when it does not detect the lower edge of the medium. The visual information recognition device according to claim 3 or 4, wherein the cropping determination unit is configured to, upon determining the third cropping state, to prohibit the visual information acquisition unit from terminating the acquisition of the visual information, and to instruct the notification unit to notify that the cropping has occurred when the operation unit has been manually operated.

6. The cropping determination unit is configured to determine the fourth cropping state, in which cropping occurs, when the visual information acquisition unit stops detecting at least one of the left or right edges of the medium after it has started acquiring the visual information. The visual information recognition device according to any one of claims 3 to 5, wherein the cropping determination unit is configured to, upon determining the fourth cropping state, to prohibit the visual information acquisition unit from terminating the acquisition of the visual information, and to instruct the notification unit to notify that cropping has occurred, until the cropping determination unit no longer determines that the fourth cropping state has occurred.

7. The cropping determination unit is configured to determine the fourth cropping state, in which cropping occurs, when the visual information acquisition unit stops detecting at least one of the left or right edges of the medium after it has started acquiring the visual information. The visual information recognition device according to any one of claims 3 to 5, wherein the cropping determination unit is configured to, upon determining the fourth cropping state, to cancel the acquisition of the visual information by the visual information acquisition unit and to instruct the notification unit to notify that cropping has occurred.

8. The cropping determination unit is configured to determine the fifth cropping state, in which cropping occurs, when the visual information acquisition unit stops detecting the medium after it has started acquiring the visual information. The visual information recognition device according to any one of claims 3 to 7, wherein the cropping determination unit is configured to, upon determining the fifth cropping state, to cancel the acquisition of the visual information by the visual information acquisition unit and to instruct the notification unit to notify that cropping has occurred.

9. A visual information recognition system that acquires moving images by continuously capturing images of a medium that displays visual information, An acquisition unit that acquires the moving image captured on the aforementioned medium, A visual information acquisition unit capable of acquiring character information from the visual information by OCR processing based on acquisition target data which is at least one of the multiple image data that constitute the moving image, The system includes a cropping determination unit that determines whether or not cropping of the medium occurs in the acquired data based on the detection state of the edge of the medium in the acquired data, A visual information recognition system in which the process of acquiring the character information by the visual information acquisition unit and the process of determining the cut-off part of the image are performed simultaneously in parallel.

10. A visual information recognition method in a visual information recognition system that acquires moving images by continuously capturing images of a medium that displays visual information, An acquisition step of acquiring the video image captured by the aforementioned medium, A visual information acquisition step in which, based on the target data which is at least one of the multiple image data that constitute the moving image, character information of the visual information is acquired by OCR processing, The system includes a cropping determination step, which determines whether or not cropping of the medium occurs in the acquired data based on the detection state of the edge of the medium in the acquired data, A visual information recognition method in which the acquisition of character information by the visual information acquisition step and the determination of whether or not the medium is cropped by the cropping determination step are performed simultaneously.

11. A visual information recognition program that acquires moving images by continuously capturing images of a medium that displays visual information, An acquisition function for acquiring the video image captured on the aforementioned medium, A visual information acquisition function that acquires character information from the visual information by OCR processing based on target data which is at least one of the multiple image data that constitute the moving image, A computer is instructed to perform a cropping detection function that determines whether or not cropping of the medium occurs in the acquired data, based on the detection state of the edge of the medium in the acquired data. A visual information recognition program configured such that the acquisition of character information by the visual information acquisition function and the determination of whether or not the media is cropped by the cropping determination function are performed simultaneously.

Images