Optical information reading device

Abstract

To lower the lower limit of PCC capable of performing decoding processing by automatically applying super-resolution processing by a neural network when the PCC is small to the extent that decoding processing cannot be performed.SOLUTION: An optical information reader includes: a storage part for storing neural network structure and parameters for outputting an enlarged image obtained by enlarging PPC for showing the number of pixels of each module for composing a code included in a read image generated by a camera; and a processor for inputting a read image generated by the camera to the neural network composed of structure and parameters stored in the storage part, generating an enlarge image obtained by enlarging a PPC value of the inputted read image, and executing decoding processing to the enlarged image.SELECTED DRAWING: Figure 19

Description

Technical Field

[0002]

[0001] The present invention relates to an optical information reading device that optically reads information.

Background Art

[0002] In recent years, so-called traceability, which enables tracking of, for example, the distribution route of goods from the manufacturing stage to the consumption stage or the disposal stage, has been regarded as important, and code readers for this traceability purpose have become widespread. In addition to traceability, code readers are also used in various fields. [[ID=I3]]

[0003] Generally, a code reader is configured to be able to photograph a code such as a barcode or a two-dimensional code attached to a workpiece with a camera, cut out the code included in the obtained image by image processing, binarize it, and perform a decoding process to read information. Since it is a device that optically reads information, it is also called an optical information reading device.

[0004] As this type of optical information reading device, for example, as disclosed in Patent Document 1, there is known one provided with a machine learning device that learns a model structure showing the relationship between an image of a code acquired by a visual sensor of a robot and an image of an ideal code. Patent Document 1 describes that by applying the learning result by the machine learning device to the image of the code acquired by the visual sensor during operation, the image is restored to an image suitable for reading.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] By the way, when reading an image of a code captured by a camera, there is a limit to the decoding process that can be performed, depending on the number of pixels (PPC) of each module that makes up the code. In order to successfully decode the code reliably, the PPC must be above a certain level, and if the PPC falls below a certain value, decoding may not be possible at all.

[0007] Here, various super-resolution processing techniques are known for increasing the resolution of PPC, or read images. As an example of application to optical information reading devices, there is a super-resolution technique that uses multiple frame images, but super-resolution processing using multiple frame images has problems such as requiring the capture of multiple frames, which takes time to process, and difficulty in aligning the frame images.

[0008] This invention has been made in view of the above, and its purpose is to lower the lower limit of PPC that allows decoding by enabling super-resolution processing to be automatically applied by a neural network when the PPC is too small to perform decoding. [Means for solving the problem]

[0009] To achieve the above objective, the first aspect of this disclosure may be based on an optical information reading device for reading a code attached to a workpiece. The optical information reading device comprises a camera that photographs the code and generates a read image, a storage unit that stores the structure and parameters of a neural network that outputs an enlarged image in which the PPC, which indicates the number of pixels of each module constituting the code included in the read image generated by the camera, is enlarged, and a processor is configured to input the read image generated by the camera to the neural network configured with the structure and parameters stored in the storage unit, generate an enlarged image in which the PPC values ​​of the input read image are enlarged, and perform a decoding process on the enlarged image.

[0010] In other words, for example, the further the camera is from the workpiece, the smaller the captured code becomes, causing the PPC to fall below the read limit. Also, in the optical system of a handheld optical information reader, for example, even if the distance is not great, the module may be small, causing the PPC to fall below the read limit. When such a read image is input to a neural network, the processor generates an enlarged image with an enlarged PPC value of the input read image. This allows decoding to be performed on the enlarged image, which has a PPC value larger than the read limit.

[0011] In a second aspect of this disclosure, the optical information reading device includes a PPC value acquisition unit that acquires the PPC value of a code contained in a reading image generated by the camera. The processor can generate the enlarged image if the PPC value acquired by the PPC value acquisition unit is less than or equal to a predetermined value, while not generating the enlarged image if the PPC value acquired by the PPC value acquisition unit exceeds the predetermined value.

[0012] With this configuration, if the PPC value is below a predetermined value, for example, below the reading limit, a magnified image is generated, enabling the decoding process. On the other hand, if the PPC value exceeds the reading limit, the decoding process can be performed without generating a magnified image, so in this case, the decoding result can be output early without processing by the neural network.

[0013] In a third aspect of this disclosure, the PPC value acquisition unit can acquire a read image generated by the camera when the optical information reading device is set up, and acquire the PPC value of the code contained in the acquired read image.

[0014] With this configuration, when setting the imaging conditions of the optical information reading device, the reading image generated by the camera can be acquired, and the PPC value of the code contained in the acquired reading image can be obtained. Therefore, it is possible to determine at the time of setup whether or not it is necessary to generate an enlarged image, and if necessary, to automatically generate an enlarged image during operation.

[0015] In a fourth aspect of this disclosure, the processor may generate an enlarged image if it fails to decode the code contained in the image read by the camera when setting up the optical information reading device, and may perform a decoding process on the enlarged image.

[0016] In other words, in order to set the imaging conditions of the optical information reading device, the decoding process must be successful. If the PPC of the code in the reading image used during the setting process is below the reading limit, the setting may not be possible. In this case, the processor generates an enlarged image, which enables the decoding process, and as a result, various settings can be made.

[0017] In a fifth aspect of this disclosure, the processor may be configured to have a decoding processing unit that performs a first decoding process for decoding a read image generated by a camera and a second decoding process for decoding the enlarged image.

[0018] With this configuration, for example, if the decoding of the read image fails in the first decoding process, the decoding of the enlarged image can be performed in the second decoding process. The first and second decoding processes may be executed in parallel, or one may be executed first and then the other.

[0019] In a sixth aspect of this disclosure, the processor may have a core that performs the first decoding process and the second decoding process in parallel using separate threads.

[0020] In a seventh aspect of this disclosure, the decoding processing unit may be configured as a multicore system capable of executing the first decoding process and the second decoding process on different cores. With this configuration, the core that executes the first decoding process and the core that executes the second decoding process are on separate cores, so that the first decoding process and the second decoding process can be executed in parallel, thereby reducing processing time.

[0021] In an eighth aspect of the present disclosure, the optical information reading device includes a region extraction unit that extracts a code candidate region where a code is likely to exist from the captured image generated by the camera. The processor can input a partial image corresponding to the code candidate region extracted by the region extraction unit to the neural network to generate the enlarged image.

[0022] According to this configuration, when the optical information reading device is in operation, when a code candidate region is extracted from the captured image generated by the camera, a partial image corresponding to the code candidate region is input to the neural network to generate an enlarged image. In this case, since there are almost no cases where a code exists in the entire captured image, the size of the partial image corresponding to the code candidate region is smaller than the size of the captured image. As a result, the size of the image input to the neural network becomes smaller, so the computational load on the processor is reduced, and thus the processing speed can be increased.

[0023] In a ninth aspect of the present disclosure, the region extraction unit can search for a code in the captured image generated by the camera based on information for specifying the code, and extract a region including the searched code as the code candidate region.

[0024] According to this configuration, the optical information reading device can automatically extract the code candidate region without the user performing an operation for extracting the code candidate region, thus reducing the burden on the user.

[0025] In a tenth aspect of the present disclosure, the optical information reading device includes a light irradiation unit that irradiates visible light of a color different from the ambient light into the imaging field range of the camera to form a mark. The region extraction unit can specify a portion corresponding to the mark formed by the light irradiation unit from the captured image generated by the camera, and extract the specified portion as the code candidate region.

[0026] According to this configuration, a mark is formed by visible light within the imaging field range of the camera. For example, in an optical information reading device having a handheld housing with a gripping portion for the user to grip during operation, by the user performing an operation to align the mark with the code, the range where the code is surely included can be imaged by the camera. Therefore, in the read image generated by the camera, it is highly likely that the code exists in the portion corresponding to the mark. By extracting the portion corresponding to this mark as a code candidate region, the possibility that the extracted region contains the code is further increased, and thereby the success rate of the decoding process can be enhanced.

[0027] In an eleventh aspect of the present disclosure, the optical information reading device may further include a filter processing unit that executes a noise removal filter on the read image before inputting the read image generated by the camera into the neural network.

[0028] According to this configuration, factors that have an adverse effect during the generation of the enlarged image by the neural network can be eliminated.

Advantages of the Invention

[0029] As described above, by using a neural network that outputs an enlarged image obtained by enlarging the PPC indicating the number of pixels of each module constituting the code included in the read image, the lower limit of the PPC that can be read can be lowered. Thus, for example, decoding processing of a code photographed at a long distance or a code with a small module size becomes possible.

Brief Description of the Drawings

[0030] [Figure 1] It is a diagram for explaining the operation of a stationary optical information reading device. [Figure 2] It is a perspective view of a stationary optical information reading device. [Figure 3] It is a block diagram of an optical information reading device. [Figure 4] It is a perspective view of a handheld optical information reading device. [Figure 5] This is a schematic diagram showing the positional relationship between the optical axis of the aimer and the optical axis of the camera. [Figure 6] This diagram shows the positional relationship between the optical information reading device and the workpiece, and the aimer light and field of view. [Figure 7] This diagram illustrates the various components that make up the processor. [Figure 8] This diagram illustrates the process of extracting code candidate regions using the positional information of Aimer light. [Figure 9] This is a conceptual diagram of a neural network. [Figure 10] This flowchart shows an example of the basic steps involved in training a neural network. [Figure 11] This is a diagram showing pairs of defective and ideal images. [Figure 12] This is a conceptual diagram of a convolutional neural network used for image transformation. [Figure 13] This flowchart shows an example of the tuning procedure performed when setting up an optical information reading device. [Figure 14] This flowchart shows an example of the decoding process procedure before determining the reduction and expansion ratios. [Figure 15] This flowchart shows an example of the decoding process after determining the reduction and enlargement ratios. [Figure 16] This figure shows an example of an image read by a camera and an example of an image after applying an image processing filter. [Figure 17] This figure shows an example of an image after reduction processing and an example of an image after code region extraction. [Figure 18] This figure shows an example of performing inference processing on a scanned image using a neural network. [Figure 19] This flowchart shows the first example of operation including super-resolution processing. [Figure 20] This figure shows an example image after extracting code candidate regions from a read image and performing a filtering process. [Figure 21]This figure shows an example of generating an enlarged image from a partial image after image filtering. [Figure 22] This flowchart shows a second example of operation including super-resolution processing. [Figure 23] This flowchart shows a third example of operation including super-resolution processing. [Figure 24] This flowchart shows a fourth example of operation including super-resolution processing. [Figure 25] This is a diagram equivalent to Figure 5, showing the distinction between when super-resolution processing is performed and when it is not. [Figure 26] This is a task sequence diagram for when the reading is successful in the first decoding process. [Figure 27] This is a task sequence diagram for when the reading is successful in the second decoding process. [Figure 28] This is a task sequence diagram showing the case where reading is successful in the first and second decoding processes. [Figure 29] This is a task sequence diagram for when the second decoding process successfully reads the data twice. [Modes for carrying out the invention]

[0031] Embodiments of the present invention will be described in detail below with reference to the drawings. The following description of preferred embodiments is essentially illustrative and is not intended to limit the present invention, its applications, or its uses.

[0032] (Stationary optical information reading device) Figure 1 is a schematic diagram showing the operation of a stationary optical information reading device 1 according to an embodiment of the present invention. In this example, multiple workpieces W are placed on the upper surface of a conveyor belt B and transported in the direction of arrow Y in Figure 1, and the optical information reading device 1 according to the embodiment is installed at a distance above the workpieces W. The optical information reading device 1 is a code reader configured to photograph the code attached to the workpieces W and decode the code contained in the captured image to read the information. In the example shown in Figure 1, the optical information reading device 1 is stationary. When operating this stationary optical information reading device 1, it is fixed to a bracket or the like (not shown) to prevent it from moving. Alternatively, the stationary optical information reading device 1 may be used while being held by a robot (not shown). Furthermore, the optical information reading device 1 may be used to read the code of a stationary workpiece W. The operation of the stationary optical information reading device 1 refers to the time when it is sequentially reading the codes of the workpieces W being transported by the transport belt conveyor B.

[0033] Furthermore, each workpiece W has a code attached to its outer surface. The code includes both barcodes and two-dimensional codes. Examples of two-dimensional codes include QR code (registered trademark), micro QR code, data matrix (Data code), Veri code, Aztec code, PDF417, and Maxi code. Two-dimensional codes come in stacked and matrix types, but the present invention is applicable to any type of two-dimensional code. The code may be attached to the workpiece W by printing or engraving it directly, or by printing it on a label and then attaching it to the workpiece W; the means and method are not limited.

[0034] The optical information reading device 1 is wired to the computer 100 and the programmable logic controller (PLC) 101 by signal lines 100a and 101a, respectively. However, it is not limited to this configuration; communication modules may be built into the optical information reading device 1, the computer 100, and the PLC 101 to wirelessly connect the optical information reading device 1 to the computer 100 and the PLC 101. The PLC 101 is a control device for sequence control of the conveyor belt B and the optical information reading device 1, and a general-purpose PLC can be used. The computer 100 can be a general-purpose or dedicated electronic computer or a portable terminal.

[0035] Furthermore, during operation, the optical information reading device 1 receives a reading start trigger signal from the PLC 101 via signal line 101a, which defines the start timing for code reading. Based on this reading start trigger signal, the optical information reading device 1 performs image capture and decoding of the code. Subsequently, the decoded result is transmitted to the PLC 101 via signal line 101a. In this way, during operation of the optical information reading device 1, the input of the reading start trigger signal and the output of the decoded result are repeatedly performed between the optical information reading device 1 and an external control device such as the PLC 101 via signal line 101a. Note that the input of the reading start trigger signal and the output of the decoded result may be performed via signal line 101a between the optical information reading device 1 and the PLC 101, as described above, or via other signal lines not shown. For example, a sensor for detecting the arrival of workpiece W may be directly connected to the optical information reading device 1, and the reading start trigger signal may be input from that sensor to the optical information reading device 1.

[0036] As shown in Figure 2, the optical information reading device 1 includes a box-shaped housing 2, a polarizing filter attachment 3, an illumination unit 4, a camera 5, a display unit 6, a power connector 7, and a signal line connector 8. Furthermore, the housing 2 is provided with an indicator 9, an aimer light illumination unit (light illumination unit) 10, and operation buttons 11 and 12, and the indicator 9, aimer light illumination unit 10, and operation buttons 11 and 12 are also components of the optical information reading device 1.

[0037] The housing 2 has an elongated shape in a predetermined direction, but the shape of the housing 2 is not limited to the shape shown in the illustration. A polarizing filter attachment 3 is detachably attached to the front outer surface of the housing 2. The illumination unit 4, camera 5, aimer light irradiation unit 10, processor 20, storage device 30, ROM 40, RAM 41, etc. are housed inside the housing 2. The processor 20, storage device 30, ROM 40, and RAM 41 are also components of the optical information reading device 1.

[0038] A lighting unit 4 is provided on the front side of the housing 2. The lighting unit 4 is a part that illuminates at least the code of the workpiece W by irradiating light toward the front of the optical information reading device 1. As shown in Figure 3, the lighting unit 4 comprises a first lighting unit 4a consisting of a plurality of light-emitting diodes (LEDs), a second lighting unit 4b consisting of a plurality of light-emitting diodes, and a lighting drive unit 4c consisting of an LED driver or the like that drives the first lighting unit 4a and the second lighting unit 4b. The first lighting unit 4a and the second lighting unit 4b are driven individually by the lighting drive unit 4c and can be turned on and off separately. The lighting drive unit 4c is connected to a processor 20, and the lighting drive unit 4c is controlled by the processor 20. Note that one of the first lighting unit 4a and the second lighting unit 4b may be omitted.

[0039] As shown in Figure 2, a camera 5 is provided in the central front part of the housing 2. The optical axis direction of the camera 5 is approximately the same as the direction of light irradiation by the illumination unit 4. The camera 5 is the part that photographs the code and generates a read image. As shown in Figure 3, the camera 5 includes an image sensor 5a that receives reflected light from the code attached to the workpiece W and illuminated by the illumination unit 4, an optical system 5b having a lens and the like, and an AF module (autofocus module) 5c. Light reflected from the part of the workpiece W to which the code is attached is incident on the optical system 5b, and the incident light is emitted toward the image sensor 5a and an image is formed on the imaging surface of the image sensor 5a.

[0040] The image sensor 5a is an image sensor consisting of a light-receiving element such as a CCD (charge-coupled device) or CMOS (complementary metal oxide semiconductor) that converts the image of the code obtained through the optical system 5b into an electrical signal. The image sensor 5a is connected to the processor 20, and the electrical signal converted by the image sensor 5a is input to the processor 20 as data for the read image. The AF module 5c is a mechanism that adjusts focus by changing the position and refractive index of the focusing lens among the lenses that make up the optical system 5b. The AF module 5c is also connected to the processor 20 and controlled by the processor 20.

[0041] As shown in Figure 2, a display unit 6 is provided on the side of the housing 2. The display unit 6 consists of, for example, an organic EL display or a liquid crystal display. The display unit 6 is connected to the processor 20 and can display, for example, the code captured by the imaging unit 5, the string resulting from the decoding of the code, the reading success rate, the matching level, etc. The reading success rate is the average reading success rate when the reading process is performed multiple times. The matching level is the read margin, which indicates how easy it is to read a code that has been successfully decoded. This can be determined from the number of error corrections that occurred during decoding, etc., and can be expressed as a numerical value, for example. The fewer the error corrections, the higher the matching level (read margin), and conversely, the more error corrections, the lower the matching level (read margin).

[0042] A power cable (not shown) for supplying external power to the optical information reading device 1 is connected to the power connector 7. Signal lines 100a, 101a, etc., for communication with the computer 100 and PLC 101 are connected to the signal line connector 8. The signal line connector 8 can be configured as, for example, an Ethernet connector, a serial communication connector such as RS232C, or a USB connector.

[0043] The housing 2 is provided with an indicator 9. The indicator 9 is connected to the processor 20 and can be made up of a light-emitting element such as a light-emitting diode. The operating status of the optical information reading device 1 can be communicated externally by the illumination status of the indicator 9.

[0044] A pair of aimer light emitting units 10 are provided on the front side of the housing 2, flanking the camera 5. As shown in Figure 3, the aimer light emitting unit 10 comprises an aimer 10a made of a light-emitting diode or the like, an aimer drive unit 10b that drives the aimer 10a, and an aimer lens 10c into which the light emitted from the aimer 10a enters. The aimer 10a is used to indicate the shooting range, field of view center, and optical axis of the illumination unit 4 by emitting light (aimer light) toward the front of the optical information reading device 1, and is equipped with an element that emits light. Specifically, the aimer 10a emits visible light of a different color (e.g., red or green) from the ambient light toward the shooting field of view of the camera 5, and forms a mark visible to the naked eye on the surface onto which the visible light is emitted. The mark may be various shapes, symbols, letters, etc. The user can also set up the optical information reading device 1 by referring to the light emitted from the aimer 10a.

[0045] As shown in Figure 2, operation buttons 11 and 12 are provided on the side of the housing 2 for use when setting the optical information reading device 1, etc. Operation buttons 11 and 12 include, for example, select buttons and enter buttons. In addition to operation buttons 11 and 12, a touch panel type operation means may also be provided. Operation buttons 11 and 12 are connected to a processor 20, and the processor 20 is capable of detecting the operation status of operation buttons 11 and 12. By operating operation buttons 11 and 12, it is possible to select one of several options displayed on the display unit 6 and to confirm the selected result.

[0046] (Handheld optical information reading device) The above example shows a case where the optical information reading device 1 is stationary, but the present invention is applicable to optical information reading devices other than stationary ones. Figure 4 shows a handheld optical information reading device 1A, and the present invention can also be applied to a handheld optical information reading device 1A like the one shown in this figure.

[0047] The housing 2A of the handheld optical information reading device 1A is elongated in the vertical direction. While the orientation of the optical information reading device 1A is not limited to the illustrated orientation and can be used in various directions, for the sake of explanation, the vertical direction of the optical information reading device 1A is specified.

[0048] A display unit 6A is provided in the upper part of the housing 2A. The display unit 6A is configured similarly to the display unit 6 of the stationary optical information reading device 1. The lower part of the housing 2A is a gripping part 2B for the user to hold during operation. The gripping part 2B is a part that can be held in the same way that an average adult would hold it with one hand, and its shape and size can be freely set. By holding this gripping part 2B, the optical information reading device 1A can be carried and moved around. In other words, this optical information reading device 1A is a portable terminal device, and can also be called, for example, a handheld terminal.

[0049] Similar to the stationary optical information reading device 1, the handheld housing 2A also houses an illumination unit, camera, aimer light irradiation unit, processor, memory unit, ROM, RAM, etc. (not shown). The optical axes of the illumination unit, the camera, and the aimer light irradiation unit are directed diagonally upward from near the top end of the housing 2A. The handheld housing 2A is also equipped with a buzzer (not shown).

[0050] Multiple operation buttons 11A and a trigger key 11B are provided on or near the gripping section 2B. The operation buttons 11A are the same as the operation buttons 11 of the stationary optical information reading device 1. When the user points the tip (upper end) of the optical reading device 1A towards the workpiece W and presses the trigger key 11B, an aimer light is emitted from the tip of the optical reading device 1A, forming a mark visible to the naked eye on the surface onto which the aimer light is emitted. The user adjusts the orientation of the optical reading device 1A while visually observing the aimer light (mark) reflected from the surface of the workpiece W, and when the aimer light is aligned with the code to be read, the code reading and decoding process is performed automatically. When reading is complete, a completion notification sound is emitted from the buzzer.

[0051] One example of the use of the handheld optical information reader 1A is its application in picking operations within a logistics warehouse. For instance, when shipping ordered goods from a logistics warehouse, the necessary items are picked from the shelves within the warehouse. This picking process involves a user holding an order slip with a code, moving to the shelves, and then comparing the code on the order slip with the code attached to the product or shelf. In this case, the handheld optical information reader 1A is used to alternately read the code on the order slip and the code attached to the product or shelf.

[0052] Figure 5 is a simplified diagram showing the relationship between the optical axis of the aimer 10 of the handheld optical information reading device 1A and the optical axis of the camera 5. As shown in this figure, the optical axis of the aimer 10 and the optical axis of the camera 5 are different, so the center of the aimer light, which serves as a marker, may be far from the center of the camera 5's field of view. For example, as shown in Figure 6 as "short distance," "medium distance," and "long distance," the relative position of the center of the aimer light 10d and the center of the camera 5's field of view 5d changes depending on the distance between the handheld optical information reading device 1A and the workpiece W. Figure 6 shows the case where the aimer light 10d is cross-shaped, but it is not limited to this.

[0053] As shown in Figure 4, the handheld optical information reading device 1A can be equipped with multiple cameras 5 and 5A. One camera 5 can be equipped with an optical system for capturing images at close range, while the other camera 5A can be equipped with an optical system for capturing images at a longer distance than the other camera 5. Since cameras 5 and 5A have an autofocus function, they can generate in-focus reading images from close range to long distance.

[0054] When multiple cameras 5 and 5A are installed, the camera 5 for short distances can be a monochrome camera, and the camera 5A for long distances can be a color camera.

[0055] (Processor configuration) The following description applies to both the stationary optical information reading device 1 and the handheld optical information reading device 1A, and can be applied to either device 1 or 1A unless otherwise specified. As shown in Figure 3, the processor 20 is composed of a multi-core processor having multiple cores, which are central processing units. Specifically, the processor 20 includes a first general-purpose core 21, a second general-purpose core 22, a third general-purpose core 23, a fourth general-purpose core 24, and a dedicated core 25 dedicated to inference processing using a neural network. The first to fourth general-purpose cores 21 to 24 and the dedicated core 25 are a so-called System-on-a-chip (SoC, SOC) and are mounted on the same substrate. Note that the first to fourth general-purpose cores 21 to 24 and the dedicated core 25 do not have to be an SoC, in which case they do not have to be mounted on the same substrate, but this case is also included in the scope of the present invention. In this embodiment, the case where there are four general-purpose cores is described, but the embodiment is not limited to this; there may be one general-purpose core, or any number of two or more (for example, 6 cores, 8 cores, etc.).

[0056] The first to fourth general-purpose cores 21 to 24 and the dedicated core 25 are connected to the same memory, RAM 41, and all of the first to fourth general-purpose cores 21 to 24 and the dedicated core 25 can access the same RAM 41. In addition, the processor 20 is connected to the same memory, ROM 40, and all of the first to fourth general-purpose cores 21 to 24 and the dedicated core 25 can access the same ROM 40.

[0057] The first to fourth general-purpose cores 21 to 24 are so-called general-purpose processors, and are responsible for performing tasks such as AF control, lighting control, camera control, extraction processing to extract candidate code regions, decoding processing for read images, and various filtering processes for read images. Specific examples of processing performed by the first to fourth general-purpose cores 21 to 24 will be described later.

[0058] On the other hand, the dedicated core 25 is a core that uses a neural network to perform super-resolution processing on the read image and inference processing to generate an ideal image corresponding to the read image. It is specialized for performing multiply-accumulate operations necessary for neural network processing at extremely high speed. The dedicated core 25 includes, for example, ICs and FPGAs. Furthermore, by applying the learning results from the neural network and performing inference processing, the read image can be restored to an image suitable for decoding. Therefore, the generation of an ideal image through inference processing is also called read image restoration. In this case, the dedicated core 25 is the part that attempts to restore the read image using a neural network.

[0059] As shown in Figure 7, the processor 20 comprises an AF control unit 20a, an imaging control unit 20b, a filter processing unit 20c, a tuning execution unit 20d, a decoding processing unit 20f, an extraction unit 20g, a reduction unit 20h, an enlargement unit 20i, an inference processing unit 20j, and a super-resolution processing unit 20k. The AF control unit 20a, imaging control unit 20b, filter processing unit 20c, tuning execution unit 20d, a decoding processing unit 20f, an extraction unit 20g, a reduction unit 20h, and an enlargement unit 20i are parts composed of the arithmetic processing of the first to fourth general-purpose cores 21 to 24. On the other hand, the inference processing unit 20j and the super-resolution processing unit 20k are parts composed of the dedicated core 25.

[0060] (Configuration of the AF control unit) The AF control unit 20a is a unit that controls the AF module 5c shown in Figure 3, and is configured to focus the optical system 5b using conventionally known contrast AF or phase-detection AF. The AF control unit 20a may be composed of the cores that become the decoding processing unit 20f and the extraction unit 20g from the first to fourth general-purpose cores 21 to 24, or it may be composed of cores other than the cores that become the decoding processing unit 20f and the extraction unit 20g.

[0061] (Configuration of the imaging control unit) The imaging control unit 20b is a unit that adjusts the gain of the camera 5, controls the light intensity of the illumination unit 4, and controls the exposure time (shutter speed) of the image sensor 5a. Here, the gain of the camera 5 is the amplification ratio (also called magnification) when the brightness of the image output from the image sensor 5a is amplified by digital image processing. The light intensity of the illumination unit 4 can be changed by separately controlling the first illumination unit 4a and the second illumination unit 4b. The gain, the light intensity of the illumination unit 4, and the exposure time are imaging conditions for the camera 5. The imaging control unit 20b may be composed of the cores that become the decoding processing unit 20f and the extraction unit 20g from the first to fourth general-purpose cores 21 to 24, or it may be composed of cores other than the cores that become the decoding processing unit 20f and the extraction unit 20g. The AF control unit 20a and the imaging control unit 20b may be composed of the same core or different cores.

[0062] (Configuration of the filter processing unit) The filter processing unit 20c is the part that performs image processing filters on the read image, and may be composed of a core that also forms the decoding processing unit 20f and the extraction unit 20g, or it may be composed of a core other than the core that forms the decoding processing unit 20f and the extraction unit 20g. The core that makes up the filter processing unit 20c may also be a DSP core.

[0063] The filter processing unit 20c performs noise reduction filters to remove noise contained in the image generated by the camera 5, contrast correction filters to correct contrast, averaging filters, etc. The image processing filters performed by the filter processing unit 20c are not limited to noise reduction filters, contrast correction filters, and averaging filters, but may include other image processing filters.

[0064] The filter processing unit 20c is configured to apply an image processing filter to the read image before the super-resolution processing and inference processing described later. Furthermore, the filter processing unit 20c is configured to apply an image processing filter to the read image before the enlargement and reduction described later. The filter processing unit 20c may also be configured to apply an image processing filter to the read image after the super-resolution processing and inference processing, or to apply an image processing filter to the read image after enlargement and reduction.

[0065] (Configuration of the tuning execution unit) The tuning execution unit 20d shown in Figure 5 is responsible for setting various conditions (tuning parameters) to make them suitable for decoding by changing imaging conditions such as the gain of the camera 5, the light intensity and exposure time of the illumination unit 4, and the image processing conditions in the filter processing unit 20c when setting up the optical information reading devices 1 and 1A. The image processing conditions in the filter processing unit 20c include the coefficients of the image processing filter (the strength of the filter), switching between image processing filters if there are multiple image processing filters, and combinations of different types of image processing filters. Appropriate imaging and image processing conditions vary depending on the influence of ambient light on the workpiece W during transport, and the color and material of the surface to which the code is attached. Therefore, the tuning execution unit 20d searches for more appropriate imaging and image processing conditions and sets the processing by the AF control unit 20a, imaging control unit 20b, and filter processing unit 20c.

[0066] Furthermore, the tuning execution unit 20d includes a PPC value acquisition unit 20n. The PPC value acquisition unit 20n is the part that acquires the PPC values ​​of codes included in the read image generated by the camera 5 when the optical information reading device 1 is set up, and is also configured to acquire the size of the codes included in the read image generated by the camera 5. When acquiring the size of a code, first the PPC value acquisition unit 20n searches for a code based on feature quantities that indicate code-likeness. Next, the tuning execution unit 20d acquires the PPC, code type, code size, etc. of the searched code. PPC, code type, code size, etc. are included in the code parameters or code conditions, and therefore the PPC value acquisition unit 20n is configured to acquire the code parameters or code conditions of the searched code.

[0067] A two-dimensional code is composed of multiple white and black modules arranged randomly. For example, in QR code Model 2, the number of modules ranges from 21x21 to 177x177. In the case of a stationary optical information reader 1, the number of modules and PPC are limited by reading the code to be read in advance, and the size of the code (pixel size) is limited to the number of modules × PPC.

[0068] PPC is a code parameter that indicates how many pixels (picture elements) a particular module is composed of, when focusing on one module among the modules that make up the code. The PPC value acquisition unit 20n can acquire the PPC by identifying a module and then counting the number of pixels that make up that module.

[0069] The code type refers to the type of code, such as QR code, data matrix code, or Vericode. Each code has its own characteristics, so the tuning execution unit 20d can determine the code type, for example, by the presence or absence of a finder pattern.

[0070] Furthermore, the code size can be calculated from the number of modules arranged vertically or horizontally in the code and the PPC. The PPC value acquisition unit 20n calculates the number of modules arranged vertically or horizontally in the code, calculates the PPC, and obtains the code size by multiplying the number of modules arranged vertically or horizontally by the PPC.

[0071] (Configuration of the decoding processing unit) The decoding unit 20f is responsible for decoding the black and white binarized data. A table showing the correspondence between encoded data can be used for decoding. Furthermore, the decoding unit 20f checks whether the decoded result is correct according to a predetermined checking method. If an error is found in the data, an error correction function is used to calculate the correct data. The error correction function varies depending on the type of code.

[0072] In this embodiment, the decoding unit 20f decodes the codes contained in the enlarged image after the super-resolution processing described later. The decoding unit 20f also decodes the codes contained in the ideal image after the inference processing. The decoding unit 20f can also decode the codes contained in the read image before super-resolution processing and the read image before inference processing. The decoding unit 20f is configured to write the decoded result obtained by decoding the codes to the decoded result storage unit 40b of the ROM 40 shown in Figure 3.

[0073] (Configuration of the extraction unit) The extraction unit 20g is responsible for extracting code candidate regions from the read image generated by the camera 5 that have a high probability of containing a code. Code candidate regions can be extracted based on features that indicate code-likeness, in which case the features that indicate code-likeness serve as information for identifying the code. For example, the extraction unit 20g can acquire a read image and search for a code in the acquired read image based on features that indicate code-likeness. Specifically, it searches whether there are any parts in the acquired read image that have a predetermined or higher level of features that indicate code-likeness, and if it can find a part with features that indicate code-likeness, it extracts the region containing that part as a code candidate region. A code candidate region may contain areas other than codes, but it will at least contain areas that have a predetermined or higher probability of being codes. It should be noted that a code candidate region is only a region that has a high probability of containing a code, so it is possible that it may not contain a code.

[0074] The extraction unit 20g can also use the positional information of the aimer light emitted from the aimer light irradiation unit 10 as information for identifying the code and extract a code candidate region. In this case, for example as shown in Figure 8, the extraction unit 20g identifies the portion corresponding to the marker formed by the aimer light 10d emitted from the aimer light irradiation unit 10 from the read image within the field of view 5d of the camera 5 by image processing, and extracts the identified portion as a code candidate region. That is, as described above, since the aimer light 10d is used to indicate the shooting range of the camera 5, the center of the field of view 5d and its vicinity, the marker formed by the aimer light 10d can be positioned in advance within the field of view 5d of the camera 5. In particular, in the case of the handheld optical information reading device 1A, the user aligns the aimer light 10d with the code to be read, so there is a high probability that the marker formed by the aimer light 10d will overlap with the code. In other words, by positioning the portion corresponding to the marker formed by the aimer light illumination unit 10 within the field of view 5d of the camera 5, when the extraction unit 20g extracts the region corresponding to the aimer light 10d within the field of view 5d of the camera 5, that region becomes a region where the code is highly likely to exist. In this case, the portion corresponding to the marker formed by the aimer light illumination unit 10 does not need to perfectly coincide with the center of the field of view of the camera 5; for example, as shown in Figure 6, it may be slightly offset in the vertical or horizontal direction of the field of view 5d. Also, since the code has a predetermined size, the region extracted by the extraction unit 20g is not only the center of the field of view 5d of the camera 5, but also a region of a predetermined size that includes the center of the field of view 5d.

[0075] As shown in Figure 6, the position of the aimer light 10d within the field of view 5d changes depending on the distance between the handheld optical information reader 1A and the code. More precisely, it changes depending on the distance between the light-receiving surface of the image sensor 5a of the camera 5 and the code. Therefore, the relationship between the distance between the optical information reader 1A and the code and the position of the aimer light 10d within the field of view 5d can be calculated in advance and stored in the storage device 30, etc. During operation, the distance between the optical information reader 1A and the code can be obtained using a well-known distance sensor or the focus information of the camera 5, and the position of the aimer light 10d within the field of view 5d can be calculated based on the obtained distance.

[0076] The extraction unit 20g can also search for a finder pattern of a code contained in the read image and extract a code candidate region based on the search result. For example, it can search for a finder pattern in a read image containing a code using image processing, determine a predetermined range of region containing that finder pattern as a code candidate region, and make that region a partial image corresponding to the code candidate region.

[0077] The extraction unit 20g can also identify the central portion of the read image and extract that central portion as a candidate code area. For example, by pre-acquiring the center of the camera 5's field of view as information for identifying the code, the portion corresponding to the center of the camera 5's field of view can be identified on the read image. In particular, in the case of the handheld optical information reading device 1A, the central portion identified on the read image corresponds to the marker formed by the aimer light irradiation unit 10, so that central portion is an area where a code is highly likely to exist.

[0078] The extraction unit 20g can be configured to accept a user's selection of a predetermined portion from the read image, and can extract the selected portion as a code candidate region. That is, when a user specifies a region of any size at any position (coordinate) on the read image, the extraction unit 20g accepts the selection of a predetermined portion from the read image based on the coordinate and size information of that position. The extraction unit 20g extracts the accepted portion as a code candidate region. For example, in the case of a stationary optical information reading device 1, as shown in Figure 1, the device photographs a workpiece W being transported by a transport belt conveyor B and performs code decoding. However, the workpiece W is not necessarily located in the center of the transport belt conveyor B in the width direction; it may be located at the edge. In this case, by specifying the portion corresponding to the edge of the transport belt conveyor B from the read image, it becomes possible to accurately extract a region that is highly likely to contain a code. In addition, in the case of a large workpiece W, the code may be displayed near the edge, away from the center. In this case, by specifying the area near the edge of the workpiece W from the read image, it becomes possible to accurately extract the region where the code is most likely to exist.

[0079] (Storage device, ROM configuration) The storage device 30 shown in Figure 3 can be configured as a read / write storage device such as an SSD (Solid State Drive). The storage device 30 can store various programs, configuration information, image data, and the like.

[0080] The ROM 40 includes an image data storage unit 40a, a decode result storage unit 40b, a parameter set storage unit 40c, and a neural network storage unit 40d. The image data storage unit 40a stores images read by the camera 5, enlarged images generated by super-resolution processing, ideal images generated by inference processing, etc. The decode result storage unit 40b stores the decoding results of the code executed by the decode processing unit 20f. The parameter set storage unit 40c stores the tuning results performed by the tuning execution unit 20d, various set conditions, code information including PPC values, and various conditions set by the user. The neural network storage unit 40d stores the structure and parameters of the trained neural network, which will be described later.

[0081] At least a portion of the above-mentioned storage units 40a, 40b, 40c, and 40d may be provided in a part other than the ROM 40, for example, in the storage device 30.

[0082] (Processing using neural networks) The optical information reading devices 1 and 1A have a magnified image generation function that generates an enlarged image by performing super-resolution processing on the image read by the camera using a neural network, and an ideal image generation function that generates an ideal image by performing inference processing on the image read by the camera 5 using a neural network. In this embodiment, the ideal image generation function is not essential.

[0083] In this embodiment, instead of performing machine learning on the optical information reading devices 1 and 1A, an example is described in which the optical information reading devices 1 and 1A pre-store the structure and parameters of a neural network that has already been trained, and perform inference processing using the neural network composed of the stored structure and parameters. However, the embodiment is not limited to this, and the structure and parameters of the neural network may also be changed by performing machine learning on the optical information reading devices 1 and 1A.

[0084] As shown in Figure 9, a neural network has an input layer into which input data (image data in this example) is input, an output layer that outputs output data, and an intermediate layer placed between the input and output layers. Multiple intermediate layers can be provided, for example, thereby creating a multi-layered neural network.

[0085] (Neural network training) First, the basic procedure for training a neural network will be explained based on the flowchart shown in Figure 10. Neural network training can be performed using a computer prepared for training purposes other than the optical information reading devices 1 and 1A, but it may also be performed using a general-purpose computer not specifically for training. Alternatively, training may be performed using the optical information reading devices 1 and 1A. Conventional and well-known methods may be used for training the neural network.

[0086] In step SA1 after the start, pre-prepared data of defective images is read. Defective images are images that have parts unsuitable for reading the code, and examples include images like those shown on the left side of Figure 11. Unsuitable parts include, for example, dirty parts or colored parts. Then, the process proceeds to step SA2, where pre-prepared data of ideal images is read. Ideal images are images that are suitable for reading the code, and examples include images like those shown on the right side of Figure 11. Defective images and ideal images are kept as pairs, and multiple such pairs are prepared. When performing learning with optical information reading devices 1 and 1A, the defective images and ideal images are input into optical information reading devices 1 and 1A.

[0087] In step SA2, the loss function is calculated to find the difference between the defective image and the ideal image. In step SA3, the neural network parameters are updated to reflect the difference found in step SA2.

[0088] Step SA4 determines whether the machine learning completion conditions are met. The machine learning completion conditions can be set based on the difference obtained in Step SA2. For example, if the difference is less than or equal to a predetermined value, it can be determined that the machine learning completion conditions are met. If Step SA4 determines YES and the machine learning completion conditions are met, the machine learning process ends. On the other hand, if Step SA4 determines NO and the machine learning completion conditions are not met, the process returns to Step SA1, loads another defective image, and then proceeds to Step SA2 to load another ideal image that is paired with the defective image in question.

[0089] In this way, a pre-trained neural network can be generated by machine learning with multiple defective images and multiple ideal images corresponding to each of those defective images. By having the optical information reading devices 1 and 1A store the structure and parameters of the pre-trained neural network, the pre-trained neural network can be constructed within the optical information reading devices 1 and 1A. As in this example, if machine learning is performed outside the optical information reading devices 1 and 1A, and the structure and parameters of the resulting neural network are stored in the optical information reading devices 1 and 1A, it becomes possible to perform inference processing using a neural network while miniaturizing and lightening the optical information reading devices 1 and 1A. Furthermore, if a processor 20 with sufficiently high processing power is installed, there is no problem in performing training within the optical information reading devices 1 and 1A.

[0090] Furthermore, when training a neural network capable of performing super-resolution processing, for example, a low-resolution image is prepared as an input image (bad image), and machine learning is performed so that a high-resolution image is output by inputting this image into the neural network. In this case, the mean squared error between the super-resolution image of the low-resolution image and the original high-resolution image (training image) can be used as the loss function of the neural network, and the parameters of the neural network are changed to reduce this error. The neural network trained in this way becomes capable of super-resolution processing, and thus becomes a neural network for super-resolution processing that can output an enlarged image in which the PPC, which indicates the number of pixels of each module that makes up the code contained in the image read by camera 5, is enlarged. A neural network for super-resolution processing can be identified by the structure and parameters of the neural network. For example, an FSRCNN can be used as a neural network for super-resolution processing.

[0091] For training a neural network for super-resolution processing, a defective image can be, for example, an image containing PPC code within a certain range. This range is, for example, PPC values ​​between 1.2 and 2.0. Furthermore, the tilt angle when capturing the defective image can be limited to approximately horizontal and vertical, but a tilt angle of approximately plus or minus 10° is also acceptable.

[0092] Figure 12 is a conceptual diagram showing an example of a Convolutional Neural Network (CNN). In the "Convolution" layer, features are extracted from the input image. This layer consists of convolutional operations, similar to image processing filters. The weights of the filter are called kernels, and feature extraction is performed according to the kernels. A convolutional layer generally has multiple different kernels, and the number of maps (D) increases according to the number of kernels. The kernel size can be, for example, 3x3 or 5x5.

[0093] In "pooling," the responses of each kernel are combined through a reduction process. In "deconvolution," the image is reconstructed from the feature values ​​using an inverse convolution filter. In "unpooling," the response values ​​are sharpened through an expansion process.

[0094] The structure and parameters of the trained neural network are stored in the neural network memory unit 40d of the ROM 40 shown in Figure 3. The structure of the neural network refers to the number of hidden layers (D) between the input layer and the output layer, the number of image processing filters, etc. The parameters of the neural network are those set in step SA3 of the flowchart shown in Figure 7.

[0095] Furthermore, the neural network memory unit 40d is capable of storing the structures and parameters of multiple neural networks with different numbers of layers or image processing filters. In this case, the structures and parameters of multiple neural networks can be stored in the neural network memory unit 40d in association with coding conditions. For example, the structure and parameters of a neural network for constituting a first neural network can be associated with a first coding condition corresponding to the first neural network. When making this association, the number of layers or filters of the first neural network can be determined in advance by the first coding condition, particularly the first PPC, so the association is made in such a way that this relationship is maintained. Similarly, the structure and parameters of a second neural network, which is different from the first neural network, can be associated with a second coding condition, which is different from the first coding condition.

[0096] When the structures and parameters of multiple neural networks are stored in the neural network memory unit 40d, the tuning execution unit 20d operates as follows when setting up the optical information reading devices 1 and 1A. Specifically, when setting the code conditions included in the read image generated by the camera 5, the tuning execution unit 20d reads the structure and parameters of the neural network associated with the set code conditions from the neural network memory unit 40d and identifies it as the neural network to be used during operation. For example, if the tuning execution unit 20d sets a first code condition as the code condition included in the read image, it reads the structure and parameters of the first neural network associated with the first code condition from the neural network memory unit 40d. Then, it identifies the structure and parameters of the first neural network as the neural network to be used during the operation of the optical information reading devices 1 and 1A. This allows inference processing to be performed on the read image with the neural network structure and parameters that are optimal for the code conditions, thereby improving reading accuracy.

[0097] When training a neural network, it is also possible to train it using defective and ideal images where the pixel resolution of the modules that make up the code is within a specific range. The pixel resolution of a module can be represented, for example, by the number of pixels (PPC) of one module that makes up the code, and machine learning of the neural network can be performed only on defective and ideal images where the PPC is within a specific range.

[0098] The specified range described above includes pixel resolutions that offer high inference processing effectiveness for read images, while excluding pixel resolutions that show little improvement in inference processing effectiveness for read images. This range can be set assuming inference processing for images containing code composed of multiple modules. This allows for improved processing speed by creating a neural network structure specifically designed for inference processing of images containing code.

[0099] The specified range for the number of pixels mentioned above can be, for example, 4 PPC or more and 6 PPC or less, but it may also be less than 4 PPC or more than 6 PPC. Furthermore, by limiting the specified range to 4 PPC or more and 6 PPC or less for machine learning, it is possible to eliminate unnecessary scale variations from the images used for training and optimize the structure of the trained neural network.

[0100] (Inference processing) As shown in Figure 7, the processor 20 of the optical information reading devices 1 and 1A is equipped with an inference processing unit 20j, which is composed of a dedicated core 25. The inference processing unit 20j reads the structure and parameters of the neural network stored in the neural network storage unit 40d of the ROM 40, and uses the read neural network structure and parameters to configure a neural network for inference processing within the optical information reading devices 1 and 1A. The neural network configured within the optical information reading devices 1 and 1A is the same as the trained neural network that was trained using the procedure shown in the flowchart of Figure 10.

[0101] The inference processing unit 20j inputs the image read by the camera 5 into the neural network and performs an inference process to generate an ideal image corresponding to the read image according to the structure and parameters of the neural network stored in the neural network memory unit 40d. The ideal image generated by the inference processing unit 20j is an image generated by the inference process and is not necessarily the same as the ideal image used during training as described above, but in this embodiment the image generated by the inference processing unit 20j is also called an ideal image.

[0102] The filter processing unit 20c is configured to perform image processing filters on the image read by the camera 5 before the inference processing unit 20j inputs the image to the neural network. This allows appropriate image processing to be performed before the neural network performs inference, resulting in more accurate inference.

[0103] The inference processing unit 20j may perform inference processing on all read images to be decoded by the decoding processing unit 20f, or it may perform inference processing on only some of the read images to be decoded by the decoding processing unit 20f. For example, the decoding processing unit 20f performs decoding on the read image before performing inference processing and determines whether the decoding was successful or not. If the decoding is successful, it means that the read image did not require inference processing, and the result is output as is. On the other hand, if the decoding fails, the inference processing unit 20j performs inference processing, and the decoding processing unit 20f performs decoding on the generated ideal image.

[0104] (Super-resolution processing) As shown in Figure 7, the processor 20 of the optical information reading devices 1 and 1A is equipped with a super-resolution processing unit 20k, which is composed of a dedicated core 25. The super-resolution processing unit 20k reads the structure and parameters of the neural network stored in the neural network storage unit 40d of the ROM 40, and uses the read neural network structure and parameters to configure a neural network for super-resolution processing within the optical information reading devices 1 and 1A. The neural network for super-resolution processing configured within the optical information reading devices 1 and 1A is the same as the trained neural network that was trained on the low-resolution and high-resolution images described above.

[0105] The super-resolution processing unit 20k inputs the image read by the camera 5 into a neural network for super-resolution processing, thereby generating an enlarged image in which the PPC values ​​of the read image are expanded according to the structure and parameters of the neural network stored in the neural network memory unit 40d. In other words, the enlarged image generated by the super-resolution processing unit 20k is an image in which the neural network has estimated high-frequency components not included in the original read image, rather than interpolating between pixels, and this makes it possible to enlarge the PPC values ​​of the read image.

[0106] For example, if the PPC value of the code contained in the image read by camera 5 is less than 2 (for example, around 1.2 to 1.5), decoding may not be possible at all. However, by increasing the PPC value to 2 or more through super-resolution processing by the super-resolution processing unit 20k, decoding becomes possible. In this embodiment, even if the PPC value is 0.1, decoding becomes possible by applying super-resolution processing.

[0107] Furthermore, while decoding is barely possible when the PPC value of the code contained in the image read by camera 5 is between 1.5 and less than 2.0, the decoding process is unstable. By setting the PPC value in this range to 2.0 or higher, stable decoding becomes possible.

[0108] The enlarged PPC value can be, for example, 2.2 or higher, or 2.4 or higher. The enlarged PPC value can be set by the enlargement ratio of the neural network. The enlargement ratio of the neural network can be any integer multiple, for example, 2.0x, 3.0x, etc. If the enlargement ratio of the neural network is set to 2.0x, and the size of the image read and input to the neural network is 128x128 (pixels), the size of the output enlarged image will be 256x256 (pixels).

[0109] The super-resolution processing unit 20k is configured to generate an enlarged image when the PPC value acquired by the PPC value acquisition unit 20n is less than or equal to a predetermined value, while not generating an enlarged image when the PPC value acquired by the PPC value acquisition unit 20n exceeds a predetermined value. The predetermined value can be any value between 2.0 and 2.3, for example. As described above, if the PPC value exceeds 2, the decoding process becomes stable, so the need for an enlarged image is low, and in this case, the processing speed can be improved by not performing the super-resolution processing.

[0110] Furthermore, the handheld optical information reading device 1A shown in Figure 4 has a close-range camera 5 and a long-range camera 5A, and each camera 5 and 5A can generate in-focus reading images from close to long distances. In this case, super-resolution processing may be performed on both the reading image generated by the close-range camera 5 and the reading image generated by the long-range camera 5A. This makes it possible to generate enlarged images from reading images that include codes at a distance or small codes that are difficult to read even when photographed at close range.

[0111] Furthermore, if the camera 5 for short distances is a monochrome camera and the camera 5A for long distances is a color camera, the color image read by the camera 5A may be converted to monochrome before super-resolution processing is performed. In this case, the processor 20 will be configured to include a monochrome conversion unit.

[0112] The filter processing unit 20c is configured to perform an image processing filter on the image read by the camera 5 before inputting the image to the neural network for super-resolution processing. This allows appropriate image processing to be performed before the super-resolution processing is carried out by the neural network for super-resolution processing, resulting in sharper magnification.

[0113] The super-resolution processing unit 20k may perform super-resolution processing on all read images decoded by the decoding processing unit 20f, or it may perform super-resolution processing on only some of the read images decoded by the decoding processing unit 20f. For example, the decoding processing unit 20f performs decoding on the read image before super-resolution processing is performed and determines whether the decoding was successful or not. If the decoding is successful, it means that the read image did not require super-resolution processing, and the result is output as is. On the other hand, if decoding fails, the super-resolution processing unit 20k performs super-resolution processing, and the decoding processing unit 20f performs decoding on the generated enlarged image.

[0114] (Input of a partial image) The images input to the neural network by the inference processing unit 20j and the super-resolution processing unit 20k may be all of the images read by the camera 5. However, the larger the size of the input image, the greater the computational load on the neural network, which may lead to a decrease in processing speed. Specifically, in a Fully Convolutional Network (FCN) type neural network as shown in Figure 12, assuming the same structure, the computational load will be proportional to the input image size, and since the input image size is proportional to the square of the code size, the computational load will also be proportional to the square of the code size.

[0115] On the other hand, it is virtually impossible for the entire image read by camera 5 to contain code. Under normal operation, code is present only in a portion of the read image, and if neural network inference processing can be performed only in that portion, the reading accuracy can be improved.

[0116] Therefore, the image input to the neural network by the inference processing unit 20j and the super-resolution processing unit 20k can be a part of the image read by the camera 5, i.e., a partial image containing the code. Specifically, the partial image corresponding to the code candidate region extracted by the extraction unit 20g is stored in the image data storage unit 40a, making it accessible to the inference processing unit 20j and the super-resolution processing unit 20k. By inputting the partial image to the neural network, the inference processing unit 20j and the super-resolution processing unit 20k perform inference processing and super-resolution processing on the partial image according to the structure and parameters stored in the neural network storage unit 40d. Alternatively, the super-resolution processing by the super-resolution processing unit 20k may be performed first, followed by inference processing by the inference processing unit 20j, and then the decoding processing by the decoding processing unit 20f may be performed.

[0117] Then, the decoding processing unit 20f performs decoding on the generated ideal image or enlarged image. Furthermore, as will be described later, if decoding is performed without performing inference processing and super-resolution processing on the partial image, the decoding processing unit 20f is made able to access the image data storage unit 40a, and the decoding processing unit 20f reads the partial image stored in the image data storage unit 40a and performs decoding.

[0118] As mentioned above, since the code is rarely present throughout the entire image being read, the size of the partial image corresponding to the code candidate region will be smaller than the size of the read image. This reduces the size of the image input to the neural network, thereby lowering the computational load on processor 20 and ultimately resulting in faster processing speed. Furthermore, since the partial image corresponds to a region where the code is most likely to exist, it is possible to create an image that contains all the information necessary for reading, i.e., the entire code. By inputting this image containing the entire code into the neural network, the decrease in reading accuracy is suppressed.

[0119] The inference processing unit 20j and the super-resolution processing unit 20k can determine the input size of a partial image to the neural network based on the code size obtained by the tuning execution unit 20d. As described above, the tuning execution unit 20d can obtain the code size. The inference processing unit 20j and the super-resolution processing unit 20k increase the input size of the partial image to be input to the neural network by a predetermined amount compared to the code size obtained by the tuning execution unit 20d.

[0120] In other words, while inputting a partial image into the neural network can reduce the computational load, if, for example, the code is rotated or the position of the code cannot be precisely detected, a portion of the code in the partial image may be missing, potentially leading to a failure in the decoding process. In this embodiment, the input size to the neural network is not the same as the code size acquired by the tuning execution unit 20d, but is larger by a predetermined amount than the code size, thereby suppressing the loss of a portion of the code in the partial image. Furthermore, by making it larger by a predetermined amount than the code size, an upper limit can be placed on the input size to the neural network, thus preventing the computational load from becoming heavy. The "determined amount" may be, for example, the horizontal (or vertical) length of the partial image input to the neural network being 1.5 times or more, 2.0 times or more, or 3.0 times or more, the horizontal (or vertical) length of the code acquired by the tuning execution unit 20d.

[0121] When the inference processing unit 20j and the super-resolution processing unit 20k determine the input size of a partial image to be input to the neural network, the determination can be made based on the number of pixels in one module that makes up the code acquired by the tuning execution unit 20d and the number of modules arranged vertically or horizontally in that code.

[0122] Furthermore, the tuning execution unit 20d can set code conditions as described above. The inference processing unit 20j and the super-resolution processing unit 20k can also set the size of the read image to be input to the neural network based on the code conditions set in the tuning execution unit 20d. For example, when the inference processing unit 20j and the super-resolution processing unit 20k acquire the code size from the code conditions set in the tuning execution unit 20d, they input a sub-image that is a predetermined amount larger than the acquired code size to the neural network. If the code size is large, the size of the sub-image input to the neural network will be large, while if the code size is small, the size of the sub-image input to the neural network will be small. In other words, the size of the sub-image input to the neural network can be changed according to the code conditions.

[0123] (Zoom in / out function) In a convolutional neural network like the one shown in Figure 12, in order to adequately capture the features of the code and create an ideal image suitable for decoding, it is necessary to extract features within a range that covers a certain number of modules. For example, if one feature value obtained from the neural network is extracted from a range of 6x6 modules, and the image is captured with one module being 20 pixels, then the neural network must be designed to calculate the aggregated feature value from a range of 120x120 pixels. In other words, the wider the pixel range to be covered, the deeper the neural network must be. For example, to aggregate feature values ​​from the aforementioned 120x120 pixel range, six convolutional layers are required.

[0124] However, since the code is composed of randomly arranged white and black modules, the relationships between pixel values ​​over a wide range are weak, and the inference processing effect hardly improves even if the covered pixel range is expanded beyond a certain point. In this specification, such characteristics are referred to as narrow-range features.

[0125] Furthermore, while the arrangement of fixed module patterns, such as finder patterns, within the code, and the fact that the code as a whole is rectangular, can be considered broad-range features, if the goal is simply to roughly separate modules from the background, inference processing can be sufficiently performed using only narrow-range features without the need for broad-range features. Broad-range features can be defined as features that have a fixed shape over a wide area.

[0126] Therefore, when training the neural network shown in Figure 10, images in which the PPC of the code is within a specific range can be used. This improves processing speed by creating a neural network structure specialized for inference processing on images containing the code, but there is a concern that the inference processing effectiveness for read images in which the PPC is outside the specific range will decrease. Note that when training the neural network, images in which the PPC of the code is outside the specific range may also be used.

[0127] In the optical information reading devices 1 and 1A of this embodiment, a function for reducing and enlarging the read image is provided to ensure that the PPC of the read image falls within a specific range. As shown in Figure 7, the processor 20 is configured with a reduction unit 20h and an enlargement unit 20i. The reduction unit 20h is the part that generates a read image reduced so that the pixel resolution of the modules constituting the code in the read image generated by the camera 5 falls within the specified range described above, and the enlargement unit 20i is the part that generates a read image enlarged so that the pixel resolution of the modules constituting the code in the read image generated by the camera 5 falls within the specified range described above.

[0128] The reduction unit 20h and the enlargement unit 20i determine, for example, whether the PPC of the code in the image read by the camera 5 is outside a specific range (4 PPC or more and 6 PPC or less). If it is within the specific range, reduction or enlargement is not performed; if it is outside the specific range, reduction or enlargement is performed. By performing reduction or enlargement, the PPC of the code in the partial image that the inference processing unit 20j and the super-resolution processing unit 20k input to the neural network falls within the specific range.

[0129] The inference processing unit 20j and the super-resolution processing unit 20k input the enlarged or reduced read image to fit within a specific range into a neural network configured with the structure and parameters stored in the neural network memory unit 40d, and perform inference processing on the read image according to that structure and parameters. Then, the decoding processing unit 20f performs decoding processing on the generated ideal image.

[0130] The extraction unit 20g may be configured to search for codes in the read image enlarged by the enlargement unit 20i or reduced by the reduction unit 20h, and to extract the region containing the searched code as a code candidate region where the code is likely to exist. In this case, the inference processing unit 20j and the super-resolution processing unit 20k input the partial image corresponding to the code candidate region extracted by the extraction unit 20g into a neural network configured with the structure and parameters stored in the neural network storage unit 40d, and perform inference processing on the read image according to that structure and parameters.

[0131] When setting up the optical information reading devices 1 and 1A, the tuning execution unit 20d can generate multiple reading images (magnified images) with different magnification ratios, and multiple reading images (reduced images) with different reduction ratios. The tuning execution unit 20d inputs the generated multiple reading images (magnified or reduced images) into a neural network to perform inference processing on each reading image, performs decoding processing on each generated ideal image, and determines the read margin, which indicates how easy it is to read the code. The tuning execution unit 20d identifies the magnification or reduction ratio of the reading image whose determined read margin is higher than a predetermined value as the magnification or reduction ratio to be used during operation. When the tuning execution unit 20d has identified the magnification or reduction ratio, during operation of the optical information reading devices 1 and 1A, the reduction unit 20h and the magnification unit 20i enlarge or reduce the reading image at the magnification or reduction ratio identified by the tuning execution unit 20d.

[0132] In other words, for example, if a specific range has a certain width, changing the magnification or reduction ratio even within that specific range may change the read margin. The magnification or reduction ratio of the read image during operation can be specified so that the read margin determined by the tuning execution unit 20d is higher than a predetermined value, thereby improving processing speed and reading accuracy.

[0133] The tuning execution unit 20d determines the reading margin for each read image, and then identifies the magnification or reduction ratio of the read image with the highest reading margin among the multiple read margins determined, as the magnification or reduction ratio to be used during operation. This further improves processing speed and reading accuracy.

[0134] Furthermore, by utilizing the ability to reduce and enlarge the read image using the reduction unit 20h and the enlargement unit 20i, the number of parameter sets can be increased. Multiple parameter sets with different reduction ratios by the reduction unit 20h and multiple parameter sets with different enlargement ratios by the enlargement unit 20i can be generated, and these parameter sets can be stored in the parameter set storage unit 40c of the ROM 40. For example, if the reduction ratio of the reduction unit 20h can be set to multiple values ​​such as 1 / 2, 1 / 4, and 1 / 8, a parameter set with a reduction ratio of 1 / 2, a parameter set with a reduction ratio of 1 / 4, and a parameter set with a reduction ratio of 1 / 8 can be stored in the parameter set storage unit 40c. Similarly, if the enlargement ratio of the enlargement unit 20i can be set to multiple values ​​such as 2x, 4x, and 8x, a parameter set with an enlargement ratio of 2x, a parameter set with an enlargement ratio of 4x, and a parameter set with an enlargement ratio of 8x can be stored in the parameter set storage unit 40c. Then, any one parameter set can be applied from among the multiple parameter sets stored in the parameter set storage unit 40c.

[0135] (Inference processing filter and super-resolution processing filter) The inference processing unit 20j may be a part that executes an inference processing filter that performs inference processing using a neural network. The super-resolution processing unit 20k may also be a part that executes a super-resolution processing filter that performs super-resolution processing using a neural network.

[0136] The inference processing filter is a filter that performs inference processing according to the structure and parameters stored in the neural network storage unit 40d by inputting the read image into a neural network composed of the structure and parameters stored in the neural network storage unit 40d, and performs the same function as the inference processing function described above.

[0137] Furthermore, the super-resolution processing filter is a filter that performs super-resolution processing according to the structure and parameters stored in the neural network storage unit 40d by inputting the read image into a neural network for super-resolution processing, which is composed of the structure and parameters stored in the neural network storage unit 40d, and thus performs the same function as the super-resolution processing function described above.

[0138] When enabling the execution of an inference processing filter or a super-resolution processing filter, a setting unit 20e (shown in Figure 7) for setting the inference processing filter or super-resolution processing filter can be provided. The setting unit 20e is configured to accept user settings for the inference processing filter or super-resolution processing filter. For example, when setting the optical information reading devices 1 and 1A, the setting unit 20e can be configured to generate a user interface that allows the user to select either "apply the inference processing filter" or "not apply the inference processing filter" and display it on the display unit 6, thereby accepting the user's selection. Alternatively, the setting unit 20e can be configured to generate a user interface that allows the user to select either "apply the super-resolution processing filter" or "not apply the super-resolution processing filter" when setting the optical information reading devices 1 and 1A, and display it on the display unit 6, thereby accepting the user's selection.

[0139] The setting unit 20e may be included in the tuning execution unit 20d, or it may be configured separately from the tuning execution unit 20d. If the setting unit 20e is included in the tuning execution unit 20d, it becomes possible to set parameters related to the inference processing filter or the super-resolution processing filter during tuning. Parameters related to the inference processing filter may include "apply the inference processing filter" and "do not apply the inference processing filter". "Apply the inference processing filter" means setting the inference processing filter to be executable, and "do not apply the inference processing filter" means not setting the inference processing filter.

[0140] Furthermore, parameters related to the super-resolution processing filter may include "apply super-resolution processing filter" and "do not apply super-resolution processing filter." "Apply super-resolution processing filter" means setting the super-resolution processing filter to be executable, while "do not apply super-resolution processing filter" means not setting the super-resolution processing filter.

[0141] Parameters relating to the inference processing filter or super-resolution processing filter are also information relating to the inference processing filter or super-resolution processing filter, and in this case, a parameter set containing information relating to the settings of the inference processing filter or super-resolution processing filter can be stored in the parameter set storage unit 40c. The parameter set stored in the parameter set storage unit 40c may include a first parameter set that enables the inference processing filter, a second parameter set that does not enable the inference processing filter, a third parameter set that enables the super-resolution processing filter, and a fourth parameter set that does not enable the super-resolution processing filter.

[0142] When the optical information reading devices 1 and 1A are in operation, one parameter set selected from the first, second, third, and fourth parameter sets is applied. When the first parameter set is applied, the decoding process by the decoding processing unit 20f is performed on the ideal image for which inference processing has been performed by the inference processing filter. On the other hand, when the second parameter set is applied, the decoding process by the decoding processing unit 20f is performed on the read image for which inference processing has not been performed. The same applies to the third and fourth parameter sets.

[0143] The parameter set stored in the parameter set storage unit 40c also includes items for setting the code conditions included in the image read by the camera 5. The items for setting the code conditions include PPC, code type, code size, etc., acquired by the tuning execution unit 20d. For example, one parameter set may include PPC, code type, and code size as items for setting the code conditions, the gain of the camera 5, the light intensity of the illumination unit 4, and the exposure time as imaging conditions, and the type of image processing filter, the parameters of the image processing filter, and the application items of the inference processing filter and super-resolution processing filter as items for the image processing filter applied by the filter processing unit 20c. The values ​​set by the tuning execution unit 20d may be used as is for each of these items, or the user may change them as they see fit.

[0144] (An example of the tuning process procedure) An example of the tuning process performed by the tuning execution unit 20d when setting up the optical information reading devices 1 and 1A will be specifically explained based on the flowchart shown in Figure 13. In step SB1 after the start of the flowchart shown in Figure 13, the tuning execution unit 20d controls the illumination unit 4 and the camera 5 to cause the camera 5 to generate a reading image, and the tuning execution unit 20d acquires the reading image. At this time, the presence and type of image processing filter to be executed before the decoding process, the neural network inference processing, and the decoding processing parameters for applying super-resolution processing are set to arbitrary parameters. Next, the process proceeds to step SB2, where the tuning execution unit 20d causes the decoding processing unit 20f to perform the decoding process on the acquired reading image.

[0145] After the decoding process, the process proceeds to step SB3, where the tuning execution unit 20d determines whether the decoding process in step SB2 was successful or not. If step SB3 determines NO and the decoding process in step SB2 fails, i.e., the code could not be read, the process proceeds to step SB4, where the decoding process parameter is changed to a different parameter, and then the decoding process is executed again in step SB2. For example, if the parameter does not perform super-resolution processing, it is changed to a parameter that does perform super-resolution processing, and super-resolution processing is performed on the read image before step SB2. If the decoding process fails with all decoding process parameters, this flow is terminated and the user is notified.

[0146] On the other hand, if step SB3 is determined to be YES and the decoding process in step SB2 is successful, the process proceeds to step SB5, where the tuning execution unit 20d determines the code parameters (PPC, code type, code size, etc.). Once the tuning execution unit 20d determines the code parameters, the structure and parameters of the neural network stored in the neural network storage unit 40d in association with the code parameters are also determined (step SB6). In step SB6, the neural network is constructed using the structure and parameters read from the neural network storage unit 40d.

[0147] In step SB7, the inference processing unit 20j performs inference processing by inputting the read image into the neural network configured in step SB6, and the decoding processing unit 20f performs decoding processing on the generated ideal image. Then, the process proceeds to step SB8, where the tuning execution unit 20d evaluates the read margin based on the decoding result of step SB7 and stores it temporarily.

[0148] Step SB9 determines whether the decoding process has been completed for all decoding parameters. If the result in Step SB9 is NO, and the decoding process has not been completed for all decoding parameters, the process proceeds to Step SB10, where the decoding parameters are changed to different parameters, and then to Step SB7.

[0149] On the other hand, if step SB9 is determined to be YES and the decoding process is completed for all decoding parameters, the process proceeds to step SB11. In step SB11, the tuning execution unit 20d selects the decoding parameter with the highest read margin from among all decoding parameters and determines that the selected decoding parameter will be applied during operation. In addition, during the tuning process, imaging conditions and other conditions are also set to appropriate conditions.

[0150] (Decoding procedure before determining reduction and enlargement ratios) Next, an example of the decoding process procedure before determining the reduction and enlargement ratios will be specifically explained based on the flowchart shown in Figure 14. The process identified in the flowchart shown in Figure 14 can be performed in step SB2 of the flowchart shown in Figure 13.

[0151] In step SC1, following the start of the flowchart shown in Figure 14, the tuning execution unit 20d selects an arbitrary image processing filter from among multiple image processing filters. This image processing filter is the filter executed by the filter processing unit 20c. Subsequently, in step SC2, the filter processing unit 20c executes the image processing filter selected in step SC1 on the read image.

[0152] Next, the process proceeds to step SC3 to create an image pyramid. The image pyramid consists of the original read image, an image of the original read image reduced to half its size, an image of the original read image reduced to a quarter of its size, an image of the original read image reduced to an eighth of its size, and so on. The reduction of the read image is performed by the reduction unit 20h. Alternatively, the image pyramid can also consist of the original read image, an image of the original read image enlarged to twice its size, an image of the original read image enlarged to four times its size, an image of the original read image enlarged to eight times its size, and so on. The enlargement of the read image is performed by the enlargement unit 20i. Note that either the reduced image or the enlarged image may be omitted.

[0153] After creating the image pyramid, the process proceeds to step SC4, where an arbitrary image is selected from among the multiple images that make up the image pyramid. This selected image may include the original image that has not been reduced or enlarged. In step SC5, the extraction unit 20g extracts code candidate regions from the image selected in step SC4 that are highly likely to contain a code. Then, the process proceeds to step SC6, where, if the parameter is one in which super-resolution processing is performed, the super-resolution processing unit 20k inputs the subimages corresponding to the code candidate regions extracted in step SC5 into the neural network and performs super-resolution processing. Also, in step SC6, if the parameter is one in which inference processing is performed, the inference processing unit 20j inputs the subimages corresponding to the code candidate regions extracted in step SC5 into the neural network and performs inference processing. The size of the subimages input to the neural network is determined by the code conditions described above. Note that if the parameter is not one in which super-resolution processing is performed, super-resolution processing is not performed in step SC6, and if the parameter is not one in which inference processing is performed, inference processing is not performed in step SC6.

[0154] After step SC6, the process proceeds to step SC7, where the code outline and the positioning of the modules constituting the code are performed. After the positioning process, the process proceeds to step SC8, where it is determined whether each module constituting the code is white or black. After determining whether it is white or black, the process proceeds to step SC9, where the decoding processing unit 20f performs the decoding process on the generated ideal image. In the decoding process, for example, a method of reconstructing the string from the 0-1 matrix of the modules can be employed.

[0155] Next, the process proceeds to step SC10, where the tuning execution unit 20d determines whether the decoding process in step SC9 was successful or not. If step SC10 determines NO and the decoding process in step SC9 has failed, the process proceeds to step SC11, where it is determined whether all the read images constituting the image pyramid have been selected. If step SC11 determines NO and not all the read images constituting the image pyramid have been selected, the process proceeds to step SC4, where another reduced-size read image or another enlarged read image constituting the image pyramid is selected, and the process proceeds to step SC5.

[0156] On the other hand, if step SC11 is determined to be YES and all the images constituting the image pyramid are selected, this flow is terminated and the conditions for successful decoding are stored.

[0157] (Decoding procedure after determining reduction and enlargement ratios) Next, an example of the decoding process procedure after determining the reduction and enlargement ratios will be specifically explained based on the flowchart shown in Figure 15. The processes specified in the flowchart shown in Figure 15 can be performed during step SB7 of the flowchart shown in Figure 13 and during the operation of the optical information reading devices 1 and 1A.

[0158] In step SD1, following the start of the flowchart shown in Figure 15, the filter processing unit 20c applies the image processing filter selected in step SC1 of the flowchart shown in Figure 14 to the read image. If the filter processing unit 20c applies an averaging filter to the read image shown in the upper part of Figure 16, the image after the image processing filter is applied, as shown in the lower part of Figure 16, is obtained.

[0159] Next, the process proceeds to step SD2, where the read image is scaled down or enlarged as needed. Specifically, if the PPC of the code in the read image after the image processing filter has been applied is within a specific range, scaling down or enlargement is not performed. If it is outside the specific range, scaling down or enlargement is performed to bring the PPC within the specific range. An example of a read image after scaling down is shown at the top of Figure 17.

[0160] Next, in step SD3, the extraction unit 20g extracts code candidate regions that are highly likely to contain codes from the read image that was reduced or enlarged in step SD2. An example of an image with extracted code candidate regions is shown at the bottom of Figure 17. If the image was not reduced or enlarged in step SD2, the extraction process will be performed on the original read image in step SD3.

[0161] Subsequently, the process proceeds to step SD4, where the inference processing unit 20j inputs the subimage corresponding to the code candidate region extracted in step SD3 into the neural network and performs inference processing. Figure 18 shows examples of the read image before inference processing and the ideal image after inference processing. After the generation of the ideal image, this flow ends via steps SD5 to SD7. Steps SD5 to SD7 are the same as steps SC7 to SC9 in the flowchart shown in Figure 14.

[0162] (First example of operation including super-resolution processing) A first example of performing super-resolution processing during the operation of the optical information reading devices 1 and 1A will be explained based on the flowchart shown in Figure 19. In step SE1 after the start, the extraction unit 20g extracts code candidate regions that are highly likely to contain a code from the read image generated by the camera 5. As shown in the example image in Figure 20, the region enclosed by the rectangular frame in the read image is the code candidate region, and this code candidate region is extracted as a partial image.

[0163] Next, the process proceeds to step SE2, where the filter processing unit 20c executes an image processing filter on the partial image. The image processing filter is a noise reduction filter. In particular, during long-distance shooting where the camera 5 and the code are far apart, the lighting may not reach the code, and the camera 5 increases the gain using automatic brightness control, which can increase random noise. If super-resolution processing is performed when there is a lot of random noise, the noise may be amplified, negatively affecting cell reconstruction. Therefore, by executing a noise reduction filter before performing super-resolution processing to remove excess noise, the effect of super-resolution processing can be enhanced. As a noise reduction filter, a normal averaging filter may obscure the cell information, so noise reduction using, for example, L1 sparse modeling is performed. At this time, contrast adjustment is also performed. The partial image after the image processing filter is shown at the bottom of Figure 20. The order of steps SE1 and SE2 may be reversed. That is, the code candidate region may be extracted after the image processing filter is executed.

[0164] Next, the process proceeds to step SE3, where the super-resolution processing unit 20k performs super-resolution processing on the partial image after image processing filtering. This generates an enlarged image, as shown in the lower part of Figure 21. This example shows the case where the magnification is 2x. The subsequent steps SE4 to SE6 are the same as steps SC7 to SC9 in the flowchart shown in Figure 14.

[0165] (Second example of operation including super-resolution processing) A second example of performing super-resolution processing during the operation of the optical information reading devices 1 and 1A will be explained based on the flowchart shown in Figure 22. Before explaining the second example, two types of decoding processes will be explained. As described above, in this embodiment, there are two cases: one in which the decoding processing unit 20f performs decoding on the reading image generated by the camera 5 without the super-resolution processing unit 20k performing super-resolution processing on the reading image generated by the camera 5, and another in which the super-resolution processing unit 20k performs super-resolution processing on the reading image generated by the camera 5 to generate an enlarged image, and the decoding processing unit 20f performs decoding on the generated enlarged image. The former case, i.e., decoding on the reading image, is called the first decoding process, and the latter case, i.e., decoding on the enlarged image generated by super-resolution processing, is called the second decoding process.

[0166] The first and second decoding processes are both executed by the decoding unit 20f, but they may be executed by different decoding units. That is, since the processor 20 has a dedicated core 25 and first to fourth general-purpose cores 21 to 24, it can execute the super-resolution processing by the super-resolution processing unit 20k, which is composed of the dedicated core 25, and the first decoding process by the decoding unit 20f, which is composed of at least one of the first to fourth general-purpose cores 21 to 24, in parallel and at high speed. Furthermore, the processor 20 can execute the second decoding process by the decoding unit 20f after the super-resolution processing by the super-resolution processing unit 20k is completed.

[0167] For example, the first general-purpose core 21 can perform the first decoding process, and the second general-purpose core 22 can perform the second decoding process. Specifically, while the first general-purpose core 21 performs the first decoding process, the dedicated core 25 performs the super-resolution process. Once the super-resolution process is complete, the second general-purpose core 22 performs the second decoding process without waiting for the first decoding process to finish. This allows the first and second decoding processes to be performed on separate general-purpose cores, resulting in even faster processing.

[0168] Furthermore, any one of the first to fourth general-purpose cores 21 to 24 of the processor 20 may be configured to execute the first decoding process and the second decoding process in parallel in separate threads. Specifically, for example, while the first general-purpose core 21 executes the first decoding process, the dedicated core 25 executes the super-resolution process. Once the super-resolution process is complete, the first general-purpose core 21 executes the second decoding process in a separate thread from the first decoding process, without waiting for the first decoding process to finish.

[0169] In step SF1 of the flowchart shown in Figure 22, the processor 20 acquires the image to be read. Then, in step SF2, the first decoding process is performed, that is, the decoding processing unit 20f performs the decoding process on the read image. After the first decoding process, the process proceeds to step SF3 to determine whether the reading was successful or not. If the reading is unsuccessful, the process returns to step SF2 and the first decoding process is executed again. If the reading is unsuccessful after a predetermined number of attempts (time), a timeout occurs and the decoding process is terminated. On the other hand, if it is determined in step SF3 that the reading was successful, the decoding process is terminated.

[0170] Furthermore, the process proceeds to step SF4 in parallel with the transition from step SF1 to step SF2. In step SF4, the extraction unit 20g extracts code candidate regions from the image read by the camera 5 that are highly likely to contain a code. After that, the process proceeds to step SF5, where one code candidate region is selected from the multiple code candidate regions extracted in step SF4, and the process proceeds to step SF6.

[0171] In step SF6, the super-resolution processing unit 20k performs super-resolution processing on the image read by the camera 5 to generate an enlarged image. In this example, the super-resolution processing is performed by the dedicated core 25, but it may also be performed by the general-purpose cores 21-24. The generation of the enlarged image and the first decoding process in step SF2 are performed in parallel. After generating the enlarged image in step SF6, the process proceeds to step SF7, where the second decoding process takes place, i.e., the decoding processing unit 20f performs decoding on the enlarged image generated in step SF6. After the second decoding process, the process proceeds to step SF8 to determine whether the reading was successful or not. If the reading is unsuccessful, the process returns to step SF5, selects another code candidate region, and then proceeds to steps SF6 and SF7 in order to perform the second decoding process again. If the reading is unsuccessful after a predetermined number of attempts (time), a timeout occurs and the decoding process ends. On the other hand, if it is determined in step SF8 that the reading was successful, the decoding process ends.

[0172] The process can be distinguished between proceeding to step SF2 and step SF4 based on the PPC value. For example, the processor 20 can be equipped with a determination unit that determines whether to proceed to step SF2 if the PPC value exceeds a predetermined value, and to proceed to step SF4 if the PPC value is less than or equal to the predetermined value. Specifically, the determination unit can be configured to proceed to step SF2 if the PPC value is 2.0 or greater, and to proceed to step SF4 if the PPC value is less than 2.0. This allows for decoding of a wide range of PPC codes. In particular, in the handheld optical information reader 1A, the distance to the code and the code size are often not constant, but even in such cases, this example can increase the success rate of the decoding process. Furthermore, in the stationary optical information reader 1, the degree of installation flexibility can be increased.

[0173] The optical information reading device 1 may be provided with a function to cancel steps SF4 to SF8. Steps SF4 to SF8 are steps that perform super-resolution processing and second decoding processing, but since super-resolution processing in particular takes time, the processing can be sped up by canceling steps SF4 to SF8. One way to cancel steps SF4 to SF8 is, for example, by user operation. For example, one of the parameters stored in the parameter set storage unit 40c may include a selection parameter for whether or not to perform super-resolution processing. By changing this selection parameter, the user can switch between a mode in which super-resolution processing is performed and a mode in which it is not performed. The selection parameter may be, for example, a "speed priority mode" selection parameter. If "speed priority mode" is selected by the user, the device will operate in a mode in which super-resolution processing is not performed, while if "speed priority mode" is not selected, the device will operate in a mode in which super-resolution processing is performed. Furthermore, based on the information obtained through tuning, if it is determined that super-resolution processing is unnecessary, the device may be configured to automatically cancel the super-resolution processing.

[0174] (Third example of operation including super-resolution processing) A third example of performing super-resolution processing during the operation of optical information reading devices 1 and 1A will be explained based on the flowchart shown in Figure 23. The third example is applicable when reading a code using the handheld optical information reading device 1A. As mentioned above, when reading small codes or codes at a distance with the handheld optical information reading device 1A, the code is read by relying on the aimer light. As shown in Figure 5, the aimer optical system usually has a different optical axis from the camera optical system, so as shown in Figure 6, the center of the aimer light 10d and the center of the field of view 5d of the camera 5 are often misaligned. Since the area in which super-resolution processing is performed should be small due to processing time constraints, in the third example, the area in which super-resolution processing is performed is limited by the positional relationship with the aimer light 10d.

[0175] After the flowchart shown in Figure 23 is started, when a read image is input, the region extraction unit 20g detects the position of the aimer light 10d (shown in Figure 8) in step SG1. In this step SG1, for example, a method can be applied in which a read image showing the aimer light 10d is captured by the camera 5 and the position of the aimer light 10d is detected by image processing, or a method can be applied in which the relationship between the distance between the optical information reading device 1A and the code and the position of the aimer light 10d within the field of view 5d is calculated in advance and stored in the storage device 30, and the distance between the optical information reading device 1A and the code is obtained using a well-known distance sensor or the focus information of the camera 5, and the position of the aimer light 10d within the field of view 5d is calculated based on the obtained distance.

[0176] Subsequently, the process proceeds to step SG2, where the region extraction unit 20g determines a candidate code region based on the position of the aimer light 10d. In step SG2, as shown in Figure 8, when the position of the aimer light 10d is detected, a predetermined range including its center is determined as a candidate code region, and this region is designated as the partial image corresponding to the candidate code region.

[0177] Next, the process proceeds to step SG3, where the filter processing unit 20c performs noise reduction filtering on the partial image corresponding to the code candidate region, and also performs contrast adjustment. After that, the process proceeds to step SG4, where the super-resolution processing unit 20k performs super-resolution processing on the filtered partial image to generate an enlarged image. Steps SG5 to SG7 are the same as steps SE4 to SE6 in the flowchart shown in Figure 19.

[0178] (Fourth example of operation including super-resolution processing) A fourth example of performing super-resolution processing during the operation of the optical information reading devices 1 and 1A will be explained based on the flowchart shown in Figure 24. The fourth example is applicable when reading a code using the handheld optical information reading device 1A and includes control to switch between performing and not performing super-resolution processing depending on the distance between the camera 5 and the code.

[0179] In other words, as shown in Figure 25, there are two cases when super-resolution processing is desired: long distances where the camera 5 and the code are far apart and the pixel resolution is low, and short distances when reading small codes. By measuring the distance between the camera 5 and the code and performing super-resolution processing only when it is a short distance (distance d ≤ distance d1) or a long distance (distance d ≥ distance d2), it is possible to avoid the decrease in processing speed that occurs when performing super-resolution processing at medium distances.

[0180] Steps SH1 to SH3 in the flowchart shown in Figure 24 are the same as steps SF1 to SF3 in the flowchart shown in Figure 22. Step SH4 proceeds in parallel with the transition from step SH1 to step SH2. In step SH4, the processor 20 determines whether the distance between the camera 5 and the code is at a medium distance. Specifically, as shown in Figure 25, the distance d between the light-receiving surface of the image sensor 5a of the camera 5 and the code is measured. The distance d can be obtained, for example, using a distance sensor or the focus information of the camera 5. If the distance d is less than or equal to distance d1 (short distance) or greater than or equal to distance d2 (long distance), the process proceeds to step SH5. On the other hand, if the distance d exceeds distance d1 and is less than distance d2 (medium distance), the process is determined to be at a distance where super-resolution processing is unnecessary, and the process terminates. Distances d1 and d2 can be the distances at which the PPC of the code included in the read image is less than or equal to a predetermined value.

[0181] Next, proceed to step SH5. Steps SH5 to SH9 are the same as steps SF4 to SF8 in the flowchart shown in Figure 22.

[0182] (Description of task sequence) Figure 26 is a task sequence diagram showing the case when the reading is successful in the first decoding process, which decodes the read image. This diagram shows the control task, the first decoding process task, the second decoding process task, and the super-resolution processing task. When the control task issues a read execution command 1 to the first decoding process task, the first decoding process starts. When the reading is successful in the first decoding process, a read success notification 2 is sent to the control task. Upon receiving the read success notification 2, the control task issues a processing completion command 3 to the first decoding process task, and the first decoding process task issues a processing completion notification 4 to the control task.

[0183] Furthermore, when the control task issues a read execution command 5 to the second decode processing task, the second decode processing task issues a super-resolution start command 6 to the super-resolution processing task. When the super-resolution processing is completed, the super-resolution processing task issues a super-resolution completion notification 7 to the second decode processing task. After that, the second decode processing task issues the next super-resolution start command 8 to the super-resolution processing task. Subsequently, since a processing completion notification 4 has been issued to the control task, the control task issues a processing completion command 9 to the second decode processing task. The second decode processing task issues a super-resolution termination command 10 to the super-resolution processing task, and the super-resolution processing task issues a super-resolution termination notification 11 to the second decode processing task. Next, the second decode processing task issues a processing completion notification 12 to the control task and terminates the read operation. In other words, the processor 20 is configured to terminate the super-resolution processing midway through, even if the super-resolution processing by the super-resolution processing unit 20j is not yet complete, if the decoding is successful in the first decode processing. This allows the next super-resolution processing to begin earlier.

[0184] Figure 27 is a task sequence diagram showing the case when reading is successful in the second decoding process, which decodes the enlarged image generated by the super-resolution process. The control task issues a read execution command 1 to the first decoding process task and a read execution command 2 to the second decoding process task. The second decoding process task issues a super-resolution start command 3 to the super-resolution process task. When the super-resolution process is complete, the super-resolution process task issues a super-resolution completion notification 4 to the second decoding process task. Subsequently, the second decoding process task issues the next super-resolution start command 5 to the super-resolution process task.

[0185] The second decoding task begins the second decoding process upon receiving the super-resolution completion notification 4. When the reading is successful in the second decoding process and the second decoding task issues a read success notification 6 to the control task, the control task issues a processing completion command 7 to the second decoding task. At this time, the control task also issues a processing completion command 8 to the first decoding task.

[0186] When the second decoding task receives the processing termination command 7, it issues a super-resolution termination command 9 to the super-resolution processing task, and the super-resolution processing task issues a super-resolution termination notification 10 to the second decoding task. The first decoding task also issues a processing termination notification 11 to the control task. The second decoding task also issues a processing termination notification 12 to the control task. In this example, if the decoding is successful in the second decoding process, the super-resolution processing by the super-resolution processing unit 20k can be terminated.

[0187] Figure 28 is a task sequence diagram showing the case when reading is successful in the first and second decoding processes. The control task issues a read execution command 1 to the first decoding task and a read execution command 2 to the second decoding task. The second decoding task issues a super-resolution start command 3 to the super-resolution processing task. The first decoding task starts the first decoding process. When the reading is successful in the first decoding process and the first decoding task issues a read success notification 4 to the control task, the control task issues a processing completion command 5 to the second decoding task. The second decoding task issues a processing completion notification 6 to the control task.

[0188] Meanwhile, when the super-resolution processing task completes the super-resolution process, it issues a super-resolution completion notification 7 to the second decoding processing task. The second decoding processing task then issues a next super-resolution start command 8 to the super-resolution processing task. Upon receiving the super-resolution completion notification 7, the second decoding processing task begins the second decoding process. If the reading is successful in the second decoding process and the second decoding processing task issues a read success notification 9 to the control task, the control task issues a processing completion command 10 to the second decoding processing task.

[0189] When the second decoding task receives the processing termination command 10, it issues a super-resolution termination command 11 to the super-resolution processing task, and the super-resolution processing task issues a super-resolution termination notification 12 to the second decoding task. After receiving the super-resolution termination notification 12, the second decoding task issues a processing termination notification 13 to the control task. In this example, the super-resolution processing by the super-resolution processing unit 20k can be terminated if decoding is successful in both the first and second decoding processes.

[0190] Figure 29 is a task sequence diagram for the case where two reads are successful in the second decoding process. The control task issues a read execution command 1 to the first decoding task and a read execution command 2 to the second decoding task. The second decoding task issues a super-resolution start command 3 to the super-resolution task. When the super-resolution process is complete, the super-resolution task issues a super-resolution completion notification 4 to the second decoding task. Subsequently, the second decoding task issues the next super-resolution start command 5 to the super-resolution task. Upon receiving the super-resolution completion notification 4, the second decoding task starts the second decoding process. If the read is successful in the second decoding process, the second decoding task issues a read success notification 6 to the control task.

[0191] Subsequently, the super-resolution processing task completes the second super-resolution processing and sends a second super-resolution completion notification 7 to the second decoding processing task. Upon receiving the second super-resolution completion notification 7, the second decoding processing task begins the second second decoding processing. The second second decoding processing is also successful, and the second decoding processing task sends a read success notification 8 to the control task. During this time, the first decoding processing task has not succeeded in reading, so it does not send a read success notification.

[0192] The control task issues a processing termination command 9 to the second decoding task. Upon receiving the processing termination command 9, the second decoding task issues a super-resolution termination command 10 to the super-resolution processing task, which then issues a super-resolution termination notification 11 to the second decoding task. After receiving the super-resolution termination notification 11, the second decoding task issues a processing termination notification 12 to the control task. Furthermore, when the control task issues a processing termination command 13 to the first decoding task, the first decoding task issues a processing termination notification 14 to the control task. In this example, if decoding is successful in both the first and second second decoding processes, the super-resolution processing by the super-resolution processing unit 20k can be terminated.

[0193] (Effects of the embodiment) As described above, according to this embodiment, a neural network can generate an enlarged image in which the PPC, which indicates the number of pixels of each module constituting the code contained in the read image, is magnified. This lowers the lower limit of the readable PCC, making it possible to decode codes captured at a long distance or codes with small module sizes, for example.

[0194] The embodiments described above are merely illustrative in all respects and should not be interpreted restrictively. Furthermore, any modifications or changes that fall within the equivalent scope of the claims are all within the scope of the present invention. [Industrial applicability]

[0195] As described above, the optical information reading device according to the present invention can be used, for example, to read a code attached to a workpiece. [Explanation of Symbols]

[0196] 1. Stationary optical information reading device 1A Handheld optical information reading device 5.5A Camera 10. Aimer Light Irradiation Unit (Light Irradiation Unit) 20 processors 20f Decode Processing Unit 20g area extraction part 20n PPC value acquisition unit 20k Super Resolution Processing Unit 21-24 1st-4th General Purpose Cores 25 dedicated cores

Claims

1. In an optical information reading device that reads a code attached to a workpiece, A camera that captures the code and generates a read image, A storage unit that stores the structure and parameters of a neural network that outputs an enlarged image obtained by expanding the PPC value, which indicates the number of pixels of each module constituting the code contained in the image read by the camera, A processor comprising a dedicated core and a general-purpose core, which controls the imaging of the camera and performs decoding processing on the enlarged image, The dedicated core reads the structure and parameters of the neural network stored in the memory unit, constructs a neural network using the structure and parameters, inputs the image read by the camera into the neural network to generate an enlarged image with enlarged PPC values, The general-purpose core is an optical information reading device that performs image acquisition control of the camera.

2. In the optical information reading device according to claim 1, The general-purpose core is an optical information reading device that outputs the result of the decoding process.

3. In an optical information reading device that reads a code attached to a workpiece, A camera that captures the code and generates a read image, A storage unit that stores the structure and parameters of a neural network that outputs an enlarged image obtained by expanding the PPC value, which indicates the number of pixels of each module constituting the code contained in the image read by the camera, A processor comprising a dedicated core and a general-purpose core, which performs decoding processing on the enlarged image, The dedicated core reads the structure and parameters of the neural network stored in the memory unit, constructs a neural network using the structure and parameters, inputs the image read by the camera into the neural network to generate an enlarged image with enlarged PPC values, The general-purpose core is an optical information reading device that outputs the result of the decoding process.

4. In the optical information reading device according to claim 2 or 3, An optical information reading device in which, in parallel with the process by which the dedicated core generates a first enlarged image, the general-purpose core is configured to output at least one of the results of decoding the read image or the results of decoding an enlarged image generated before the first enlarged image.

5. In the optical information reading device according to claim 4, The decoding process by the processor includes a first decoding process for decoding the image read by the camera and a second decoding process for decoding the first enlarged image generated by the dedicated core. An optical information reading device configured to perform in parallel the process of generating a second enlarged image using the dedicated core and the first and second decoding processes.

6. In an optical information reading device that reads a code attached to a workpiece, A camera that captures the code and generates a read image, A storage unit that stores the structure and parameters of a neural network that outputs an enlarged image obtained by expanding the PPC value, which indicates the number of pixels of each module constituting the code contained in the image read by the camera, A processor that inputs the image read by the camera to a neural network composed of the structure and parameters stored in the memory unit, generates an enlarged image by expanding the PPC value of the input read image, and performs a decoding process on the enlarged image, The optical information reading device is configured to acquire a read image generated by the camera, and a PPC value acquisition unit acquires the PPC value of the code contained in the acquired read image. The processor generates the enlarged image when the PPC value acquired by the PPC value acquisition unit is less than or equal to a predetermined value, while not generating the enlarged image when the PPC value acquired by the PPC value acquisition unit exceeds the predetermined value.

7. In the optical information reading device according to claim 6, The processor generates the magnified image when the decoding process of the code contained in the image read by the camera during the setup of the optical information reading device fails, and the optical information reading device performs the decoding process on the magnified image.

8. An optical information reading device according to any one of claims 1 to 6, The system includes a region extraction unit that searches for a code based on feature quantities for identifying a code from the image read by the camera, and extracts the region containing the searched code as a code candidate region. The processor is an optical information reading device that inputs a partial image corresponding to a code candidate region extracted by the region extraction unit into the neural network to generate the enlarged image.

9. In an optical information reading device that reads a code attached to a workpiece, A camera that captures the code and generates a read image, A storage unit that stores the structure and parameters of a neural network that outputs an enlarged image obtained by expanding the PPC value, which indicates the number of pixels of each module constituting the code contained in the image read by the camera, A processor that inputs the image read by the camera to a neural network composed of the structure and parameters stored in the memory unit, generates an enlarged image by expanding the PPC value of the input read image, and performs a decoding process on the enlarged image, A light irradiation unit for irradiating visible light of a different color from ambient light into the camera's field of view to form a marker, The system includes a region extraction unit that identifies a portion corresponding to a marker formed by the light irradiation unit from the image read by the camera, and extracts the identified portion as a candidate code region. The processor is an optical information reading device that inputs a partial image corresponding to a code candidate region extracted by the region extraction unit into the neural network to generate the enlarged image.

10. An optical information reading device according to any one of claims 1 to 5, claim 8 which is dependent on any one of claims 1 to 5, and claim 9, The system includes a PPC value acquisition unit that acquires the PPC value of the code contained in the image read by the camera, The processor generates the enlarged image when the PPC value acquired by the PPC value acquisition unit is less than or equal to a predetermined value, while not generating the enlarged image when the PPC value acquired by the PPC value acquisition unit exceeds the predetermined value.

11. An optical information reading device according to any one of claims 1 to 10, The processor is an optical information reading device having a decoding processing unit that performs a first decoding process for decoding a read image generated by a camera and a second decoding process for decoding the enlarged image.

12. In the optical information reading device according to claim 11, The processor is an optical information reading device having a core that executes the first decoding process and the second decoding process in parallel using separate threads.

13. In the optical information reading device according to claim 11, The decoding processing unit is an optical information reading device configured as a multicore capable of executing the first decoding process and the second decoding process on different cores.

14. An optical information reading device according to any one of claims 1 to 13, An optical information reading device further comprising a filter processing unit that performs a noise reduction filter on the read image generated by the camera before inputting the read image to the neural network.

15. An optical information reading device according to any one of claims 1 to 14, The aforementioned neural network is pre-trained using low-resolution images as defective images and the original high-resolution images of the defective images as training images, and the storage unit stores the structure and parameters of the pre-trained neural network in an optical information reading device.

16. An optical information reading device according to any one of claims 1 to 15, An optical information reading device further comprising a housing that accommodates the processor and the camera.

Images