Optical information reading device
By using a pre-generated neural network structure and parameters in the optical information reading device, combined with image restoration filtering and noise removal filtering, the problem of excessive computational load is solved, and the device is made lightweight and high-speed.
Patent Information
- Application Number
- CN202210026382.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-10-05
- Filing Date
- 2022-01-11
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-01-11
AI Technical Summary
Existing optical information reading devices have excessive computational load when combined with machine learning devices, making it difficult to reduce the size and weight of the device and reducing the processing speed, making them unsuitable for handheld and fixed optical information reading devices.
Using a pre-generated neural network structure and parameters, code images captured by a camera are restored, reducing the computational load on the processor. Image restoration filtering and noise removal filtering are used to improve reading accuracy, and the combination of image restoration filtering and noise removal filtering is adapted to different usage environments.
By reducing computational load while improving reading accuracy, this device achieves lightweight design for handheld optical information reading devices and high-speed processing for fixed optical information reading devices.
Smart Images

Figure CN114818763B_ABST
Abstract
Description
Technical Field
[0001] This aspect relates to an optical information reading device for reading information in an optical manner. Background Technology
[0002] In recent years, for example, the importance of so-called traceability—which enables tracking the delivery route of goods from the manufacturing stage to the consumption stage or the disposal stage—has increased, and barcode readers for this traceability purpose have been widely used. Besides traceability, barcode readers are also used in various other fields.
[0003] Typically, a code reader is configured to capture an image of a code (such as a barcode or QR code) attached to a workpiece using a camera, specify and binarize the code included in the acquired image through image processing, and perform decoding processing to read the information. Since the code reader is a device for reading information in an optical manner, it is also called an optical information reading device.
[0004] As an optical information reading device of this type, devices including a machine learning device (e.g., disclosed in Japanese Patent 6,683,666) are known, which is used to learn a model structure representing the relationship between an image of a code acquired by a robot's vision sensor and an image of an ideal code.
[0005] Japanese Patent 6683666 describes recovering an image suitable for reading by applying the learning results of a machine learning device to an image of a code acquired by a vision sensor during operation.
[0006] On the other hand, optical information reading devices include handheld optical information reading devices and fixed optical information reading devices. The handheld optical information reading device can perform code reading operations while held by a user, while the fixed optical information reading device is used when installed in a factory production line or similar facility. Considering the operability during reading, the handheld optical information reading device needs to be as small and lightweight as possible. In the fixed optical information reading device, there is a strong demand for further improvements in processing speed to handle the high-speed flow of workpieces through the production line.
[0007] However, when attempting to incorporate a machine learning device, such as that disclosed in Japanese Patent 6683666, into an optical information reading device, the computational load of image processing using machine learning is enormous, making it difficult to simplify the processor. Consequently, it is difficult to reduce the size and weight of handheld optical information reading devices. Furthermore, the enormous computational load leads to a decrease in processing speed, making it difficult to apply this machine learning device to fixed optical information reading devices. Summary of the Invention
[0008] This approach is proposed in view of these points, and its purpose is to improve reading accuracy by enabling the use of machine learning to recover code images while reducing the computational load in optical information reading devices.
[0009] To achieve the above objectives, according to one embodiment of the present invention, an optical information reading device for reading codes attached to a workpiece is used as a prerequisite. The optical information reading device includes: a camera for capturing an image of the code and generating a code image; a storage unit for storing the structure and parameters of a neural network pre-generated by machine learning on multiple defect images having portions unsuitable for code reading and multiple ideal images corresponding to the multiple defect images; and a processor for inputting the code image generated by the camera into the neural network configured using the structure and parameters stored in the storage unit to attempt to recover the code image based on the structure and parameters stored in the storage unit, and performing decoding processing on the recovered code image. The optical information reading device can be configured by housing the camera, storage unit, and processor within a housing.
[0010] According to this configuration, during the operation of the optical information reading device, when the code image generated by the camera is input into the neural network, an attempt is made to recover the code image. Therefore, a code image obtained after the recovery attempt can be obtained, which has been recovered to be suitable for code reading. Since the code image obtained after the recovery attempt is decoded, the reading accuracy is improved.
[0011] By configuring the neural network using structures and parameters obtained through pre-processing machine learning on multiple defective and ideal images, the optical information reading device only needs to perform inference processing via the neural network without undergoing learning processing, thus reducing the computational load on the processor. As a result, the size and weight of handheld optical information reading devices can be reduced, and the processing speed of stationary optical information reading devices can be improved.
[0012] According to another embodiment of the present invention, an optical information reading device for reading codes attached to a workpiece is used as a prerequisite. The optical information reading device includes: a camera for capturing an image of the code and generating a code image; a storage unit for storing the structure and parameters of a neural network pre-generated by machine learning on multiple defect images having portions unsuitable for code reading and multiple ideal images corresponding to the multiple defect images; a setting unit configured to input the code image generated by the camera into the neural network configured using the structure and parameters stored in the storage unit to set an image recovery filter for attempting to recover the code image based on the structure and parameters stored in the storage unit; and a processor for applying the image recovery filter set by the setting unit to the code image generated by the camera to attempt to recover the code image, and performing decoding processing on the code image obtained after the recovery attempt. The optical information reading device can be configured by housing the camera, storage unit, and processor within a housing.
[0013] According to this configuration, when the image restoration filter is set by the setting unit, a code image after the restoration attempt that has been restored by applying the image restoration filter is obtained, and decoding processing is performed on the code image after the restoration attempt, thereby improving the reading accuracy.
[0014] In the processing performed by image restoration filtering, the code image generated by the camera is input into a neural network to attempt to recover the code image. Because the neural network is configured using the structure and parameters obtained through pre-processing machine learning on multiple defective images and multiple ideal images, the computational load on the processor is reduced, as described above.
[0015] According to another embodiment of the present invention, the setting unit is configured to receive user settings for image restoration filtering. Therefore, the user can set image restoration filtering when it is desired to apply it, and not set it when it is not desired. That is, for example, depending on the usage environment of the optical information reading device, it is conceivable that few code images unsuitable for reading will be generated, and when the user pre-sets the option not to set image restoration filtering in such an environment, the computational load on the processor can be further reduced and the processing speed can be improved.
[0016] According to another embodiment of the present invention, a filtering processing unit is also provided, which performs image processing filtering on the code image generated by the camera before it is input into the neural network. Therefore, appropriate image processing can be performed before attempting to restore the code image using a neural network. As a result, the restoration of the code image becomes more accurate. Examples of image processing filtering include noise removal filtering, etc.
[0017] According to another embodiment of the present invention, the filtering processing unit performs noise removal filtering to remove noise included in the image generated by the camera. Therefore, for example, when attempting to recover using a neural network, the units forming the code can be successfully recovered.
[0018] According to another embodiment of the present invention, the optical information reading device may further include an adjustment execution unit for setting various conditions when setting the optical information reading device, such that the various conditions are suitable for decoding processing. The adjustment execution unit may include a setting unit configured to set an image restoration filter.
[0019] This configuration allows for adjustment of actuator settings, such as camera shooting conditions, and also enables the setting of image restoration filtering. Therefore, the application or non-application of image restoration filtering can be selected based on the user's operating environment of the optical information reading device.
[0020] According to another embodiment of the present invention, the adjustment execution unit can set parameters related to image restoration filtering. Therefore, image restoration can be performed appropriately according to the environment in which the user operates the optical information reading device.
[0021] According to another embodiment of the present invention, the adjustment execution unit may set code conditions included in the code image generated by the camera. Therefore, image restoration can be performed appropriately based on these code conditions.
[0022] According to another embodiment of the present invention, the size of the code image to be input into the neural network can be set based on the code conditions set by the adjustment execution unit. Therefore, the computational load on the processor can be reduced without reducing the reading accuracy.
[0023] According to another embodiment of the present invention, the storage unit can be configured to store a parameter set including information related to the setting of image restoration filtering. The storage unit can store a first parameter set for setting image restoration filtering and a second parameter set for not setting image restoration filtering, and can apply a parameter set selected from the first parameter set and the second parameter set to perform the decoding processing of the processor.
[0024] In other words, depending on the environment in which the user operates the optical information reading device, there may be situations where a recovery code image is not needed. In this case, by applying the second parameter set, the attempt to recover the code image is not made, thereby reducing the computational load on the processor and increasing the processing speed. On the other hand, when a recovery code image is needed, the first parameter set is applied to attempt to recover the image using a neural network, thereby enabling high-precision reading.
[0025] According to another embodiment of the invention, the processor includes an integrated circuit dedicated to attempting to recover the image. Therefore, processing can be further accelerated.
[0026] According to another embodiment of the invention, the housing is a handheld housing having a grip portion held by the user during operation. Therefore, while the user holds the housing, an image of the code can be captured by pointing the camera in the desired direction. In this case, the user can switch between applying and not applying image restoration filtering as needed.
[0027] According to another embodiment of the present invention, the processor is configured to perform decoding processing on the code image before attempting to recover it, attempt to recover the code image when decoding fails, and perform decoding processing on the recovered code image. That is, in the case of generating a code image that does not require recovery, decoding processing is made successful by performing decoding processing on the code image before attempting recovery. Therefore, recovery, which increases computational load, is eliminated, and processing is accelerated. On the other hand, in the case of decoding failure, the code image is recovered and then subjected to decoding processing. Therefore, the success rate of decoding can be improved.
[0028] According to another embodiment of the present invention, an image restoration unit is also provided for inputting a code image into a neural network and performing inference processing to generate a restored image obtained by restoring the code image; a first decoding processing unit is provided for performing a first decoding processing on a code image generated by a camera; and a second decoding processing unit is provided for performing a second decoding processing on the restored image generated by the image restoration unit. The second decoding processing unit extracts code candidate regions from the code image and sends a trigger signal to the image restoration unit to perform inference processing. When the trigger signal sent from the second decoding processing unit is received, the image restoration unit inputs a portion of the image corresponding to the code candidate region into the neural network and performs inference processing to generate a restored image obtained by restoring the portion of the image. The second decoding processing unit can determine the grid positions of each cell position of the indicator code based on the restored image and perform decoding of the restored image based on the determined grid positions. That is, the first decoding processing unit performs decoding of a normal code image that has not undergone inference processing, the image restoration unit performs generation of a restored image obtained by inference processing, and the second decoding processing unit decodes the restored image obtained by inference processing. Therefore, processing speed can be improved by constructing processing circuits suitable for each unit. In addition, the image restoration unit only needs to restore the portion of the image corresponding to the code candidate region, thus further improving the processing speed.
[0029] According to another embodiment of the present invention, the inference processing of the image restoration unit and the first decoding processing of the first decoding processing unit are performed in parallel. Therefore, the processing speed can be improved. Furthermore, at least a portion of the time period during which the second decoding processing unit performs the second decoding processing has a pause period during which the inference processing of the image restoration unit is not performed. Therefore, the load on the image restoration unit can be reduced and heat generation suppressed. In this case, the second decoding processing unit may include multiple cores.
[0030] According to another embodiment of the present invention, the first decoding processing unit can perform the first decoding process in a shorter time than the time required for the second decoding process. Furthermore, the setting unit can switch whether to perform the second decoding process, that is, whether to perform the decoding process of the recovered image. Since the second decoding process performs the decoding process of the recovered image, the time required for this process is longer than the time required for the first decoding process, which performs the decoding process of the normally encoded image. Assuming this, when the system is set to perform the second decoding process, a decoding timeout (decoding limit time) longer than the time required for performing the second decoding process can be set. On the other hand, when the system is set not to perform the second decoding process, a decoding timeout shorter than the time required for performing the second decoding process can be set. Therefore, when the decoding of the recovered image exceeds the decoding timeout, the system can quickly switch to the decoding process of the next captured image.
[0031] According to another embodiment of the present invention, the decoding processing unit can perform a second decoding process on the recovered image in parallel with the first decoding process for the coded image generated by the camera. Therefore, even if both the coded image and the recovered image exist, the processing speed can be improved. When the setting unit is configured not to perform the second decoding process, the decoding processing unit can allocate resources intended for the second decoding process to the first decoding process. Processing resources include, for example, memory usage areas and cores forming a multi-core CPU. If the second decoding process is set not to be performed when memory areas or cores are pre-configured as processing resources for the second decoding process, the memory areas and cores can be used for the first decoding process, thus further accelerating the first decoding process.
[0032] According to another embodiment of the present invention, the decoding processing unit extracts code candidate regions with a high probability of code presence from a code image. The image restoration unit inputs a portion of the image corresponding to the code candidate region into a neural network and generates a restored image obtained by restoring that portion of the image. The decoding processing unit determines grid positions based on the restored image generated by the image restoration unit and performs decoding of the restored image based on the determined grid positions. That is, in many cases, generating the restored image takes time, and the time required to generate the restored image can be shortened by extracting code candidate regions based on features such as indicator code similarity before generating the restored image. In addition, the determination of grid positions can be performed with high accuracy based on the restored image, thus improving reading performance.
[0033] According to another embodiment of the present invention, an adjustment process is performed to determine optimal imaging and decoding conditions and to set the size of the code to be read. The decoding processing unit can determine the size of the extracted image, including the code candidate region, in the code image based on the code size set in the adjustment process. Since the extracted image is enlarged or reduced to an image of a predetermined size and then input into the neural network, the PPC of the code image input into the neural network can be kept within a predetermined range, and the image restoration effect using the neural network can be stably achieved. Furthermore, it is conceivable that various sizes of codes to be read exist, but the size of the code ultimately input into the neural network can be fixed; therefore, the balance between high-speed processing and ease of decoding can be maintained at a certain level or higher.
[0034] According to another embodiment of the invention, a camera receives light that has passed through the first polarizer and reflected from the code via a second polarizer, and generates a code image with lower contrast compared to the case where neither the first nor the second polarizer is inserted. Since the specular reflection component of the workpiece is removed by using polarizers, a code image with reduced specular reflection component effects can be obtained. For example, in the case of obtaining an image by photographing a metal workpiece, polarizers are suitable for images with many specular reflection components, but there are cases where the code image becomes darker and the contrast decreases due to reduced light intensity. In this case, reading performance is improved by utilizing inference processing of a neural network to convert the code image into a recovered image with high contrast.
[0035] According to another embodiment of the present invention, the adjustment execution unit can set multiple imaging conditions and code conditions, including a first imaging condition and a code condition, and a second imaging condition and a code condition. The decoding processing unit can input the code image generated under the first imaging condition and code condition into the neural network to generate the recovered image, and when the decoding processing of the recovered image fails, it can input the code image generated under the second imaging condition and code condition, which is different from the first imaging condition and code condition, into the neural network to generate the recovered image, and perform decoding of the recovered image.
[0036] As described above, the code image can be recovered using a neural network configured with a structure and parameters obtained through pre-processing machine learning on multiple defective images and multiple ideal images. Therefore, by enabling the recovery of the code image using machine learning while reducing the computational load on the optical information reading device, reading accuracy can be improved. Attached Figure Description
[0037] Figure 1 This is a diagram used to describe the operating time of a fixed optical information reading device;
[0038] Figure 2 This is a perspective view of a fixed optical information reading device;
[0039] Figure 3 This is a block diagram of an optical information reading device;
[0040] Figure 4 This is a perspective view of a handheld optical information reading device;
[0041] Figure 5 It is a diagram used to describe the various parts configured by the processor;
[0042] Figure 6 This is a conceptual diagram of a neural network;
[0043] Figure 7 This is a flowchart illustrating an example of the basic process during the learning of a neural network;
[0044] Figure 8A and 8B A pair of defective and ideal images are shown, in which Figure 8A This is an example of a defective image, and Figure 8B This is an example of an ideal image;
[0045] Figure 9 This is a conceptual diagram of a convolutional neural network used for image transformation;
[0046] Figure 10 This is a flowchart illustrating an example of the adjustment steps performed during the setup of the optical information reading device;
[0047] Figure 11 This is a flowchart illustrating an example of the decoding process before determining the reduction and magnification rates;
[0048] Figure 12 This is a flowchart illustrating an example of the decoding process after the reduction and magnification rates are determined;
[0049] Figure 13A This is a view showing an example of a code image generated by a camera, and Figure 13B This is a view showing an example of the image obtained after performing image processing filtering;
[0050] Figure 14A This is a view showing an example of the image obtained after the downsizing process, and Figure 14B This is a view showing an example of the image obtained after code region extraction;
[0051] Figure 15 This is a view showing an example of code image recovery obtained using a neural network;
[0052] Figure 16 This is a diagram illustrating an example of a decoding process according to another embodiment;
[0053] Figure 17 This is a timing diagram of the decoding process performed by the first decoding processing unit and the second decoding processing unit;
[0054] Figure 18 This is a timing diagram of the decoding process performed by the decoding processing unit;
[0055] Figure 19 This is a flowchart illustrating an example of the first and second decoding processes;
[0056] Figure 20 This is a view showing example images of each process;
[0057] Figures 21A to 21C It is a view used to describe the use of multiple neural networks to deal with the black-and-white reversal of codes;
[0058] Figures 22A to 22C It is a view used to describe the use of a neural network to deal with the black-and-white reversal of codes;
[0059] Figure 23 It is a graph showing the relationship between contrast and matching level;
[0060] Figure 24 This is a flowchart illustrating an example of the processing used to calculate the matching level;
[0061] Figure 25This is a view showing an example of a user interface displayed on the display unit; and
[0062] Figures 26A to 26C This is a diagram illustrating an example of using an imaging element with an AI chip. Detailed Implementation
[0063] In the following description, embodiments of this aspect will be described in detail with reference to the accompanying drawings. Note that the following description of preferred embodiments is merely illustrative in nature and is not intended to limit this aspect, its application, or its use.
[0064] (Fixed optical information reading device)
[0065] Figure 1 This is a schematic diagram illustrating the operation time of the fixed optical information reading device 1 according to an embodiment of this aspect. In this example, multiple workpieces W are placed on the upper surface of the conveyor belt type conveyor section B along... Figure 1 The optical information reading device 1, according to this embodiment, is installed at a position where it separates upwards from the workpiece W, conveying information in the direction of arrow Y. The optical information reading device 1 is a code reader configured to capture an image of a code attached to the workpiece W and read information by decoding the code included in the captured image. Figure 1 In the example shown, the optical information reading device 1 is a fixed type. During the operation time of the fixed optical information reading device 1, it is used in a state where it is fixed to a bracket or the like (not shown) so that it does not move. Note that the fixed optical information reading device 1 can also be used in a state where it is held by a robot (not shown). In addition, the code of the workpiece W in the fixed state can be read by the optical information reading device 1. The operation time of the fixed optical information reading device 1 is the time for performing the operation of sequentially reading the code of the workpiece W conveyed by the conveyor belt conveyor B.
[0066] Additionally, codes are attached to the outer surface of each workpiece W. Codes include both barcodes and QR codes. Examples of QR codes include QR codes (registered trademark), Micro QR codes, data matrices (data codes), Veri codes, Aztec codes, PDF 417, and Maxi codes. QR codes can be stacked or matrix-type, and this aspect can be applied to any QR code. Codes can be directly attached to the workpiece W by printing or engraving, or by pasting them onto the workpiece W after printing on a label, and any mechanism or method can be used.
[0067] The optical information reading device 1 is wired to the computer 100 and the programmable logic controller (PLC) 101 via signal lines 100a and 101a, respectively. However, this is not a limitation, and the optical information reading device 1, the computer 100, and the PLC 101 may have built-in communication modules to wirelessly connect the optical information reading device 1 to the computer 100 and the PLC 101. The PLC 101 is a control device configured for the sequential control of the conveyor belt conveyor unit B and the optical information reading device 1, and a general-purpose PLC can be used. The computer 100 can be a general-purpose or dedicated electronic calculator or a portable terminal, etc.
[0068] Additionally, during its operation time, the optical information reading device 1 receives a read start trigger signal from the PLC 101 via signal line 101a, which defines the timing for starting code reading. Then, the optical information reading device 1 captures an image of the code and decodes the image based on the read start trigger signal. The decoding result is then sent to the PLC 101 via signal line 101a. In this way, during the operation time of the optical information reading device 1, the input of the read start trigger signal and the output of the decoding result are repeatedly performed via signal line 101a between the optical information reading device 1 and an external control device such as the PLC 101. Note that, as described above, the input of the read start trigger signal and the output of the decoding result can be performed via signal line 101a between the optical information reading device 1 and the PLC 101, or via another signal line (not shown). For example, a sensor configured to detect the arrival of workpiece W and the optical information reading device 1 can be directly connected to each other so that the read start trigger signal is input from the sensor to the optical information reading device 1.
[0069] like Figure 2 As shown, the optical information reading device 1 includes a box-shaped housing 2, a polarizing filter accessory 3, an illumination unit 4, a camera 5, a display unit 6, a power connector 7, and a signal line connector 8. Furthermore, the housing 2 is equipped with an indicator 9, a sight illumination unit 10, and operation buttons 11 and 12, and the indicator 9, the sight illumination unit 10, and the operation buttons 11 and 12 are also components of the optical information reading device 1.
[0070] The housing 2 has an elongated shape in a predetermined direction, but the shape of the housing 2 is not limited to the shape shown. A polarizing filter attachment 3 is detachably attached to the front outer surface of the housing 2. The housing 2 houses an illumination unit 4, a camera 5, a sight illumination unit 10, a processor 20, a storage unit 30, a ROM 40, and a RAM 41, etc. The processor 20, storage unit 30, ROM 40, and RAM 41 are also components of the optical information reading device 1.
[0071] The illumination unit 4 is disposed on the front side of the housing 2. The illumination unit 4 is configured to illuminate at least the code on the workpiece W by emitting light towards the front of the optical information reading device 1. Furthermore... Figure 3 As shown, the lighting unit 4 includes a first lighting unit 4a formed by a plurality of light-emitting diodes (LEDs), a second lighting unit 4b formed by a plurality of light-emitting diodes, and a lighting driving unit 4c formed by an LED driver for driving the first lighting unit 4a and the second lighting unit 4b. The first lighting unit 4a and the second lighting unit 4b are each driven by the lighting driving unit 4c and can be turned on and off independently. The lighting driving unit 4c is connected to the processor 20 and is controlled by the processor 20. Note that either the first lighting unit 4a or the second lighting unit 4b may be omitted.
[0072] like Figure 2 As shown, camera 5 is disposed in the central part of the front side of housing 2. The optical axis of camera 5 is substantially aligned with the light illumination direction of illumination unit 4. Camera 5 is the part used to capture codes and generate code images. Figure 3 As shown, the camera 5 includes: an imaging element 5a for receiving light reflected from a code attached to the workpiece W and illuminated by an illumination unit 4; an optical system 5b having lenses, etc.; and an autofocus module (AF module) 5c. Light reflected from the portion of the workpiece W to which the code is attached is incident on the optical system 5b, and the incident light is emitted toward the imaging element 5a and forms an image on the imaging surface of the imaging element 5a.
[0073] Imaging element 5a is an image sensor that includes light-receiving elements such as charge-coupled devices (CCDs) and complementary metal-oxide-semiconductor (CMOS). This image sensor converts the image of the code obtained by the optical system 5b into an electrical signal. Imaging element 5a is connected to processor 20, and the electrical signal converted by imaging element 5a is input to processor 20 as data of the code image. Additionally, AF module 5c is a mechanism that performs focusing by changing the position and refractive index of a focusing lens in the lens constituting optical system 5b. AF module 5c is also connected to and controlled by processor 20.
[0074] like Figure 2As shown, the display unit 6 is disposed on the side of the housing 2. The display unit 6 is configured using, for example, an organic EL display or a liquid crystal display. The display unit 6 is connected to the processor 20 and can display, for example, a code captured by the camera 5, a string as the decoding result of the code, a read success rate, a match level, etc. The read success rate is the average read success rate when read processing is performed multiple times. The match level is a read margin indicating the ease of reading a code that has been successfully decoded. This can be obtained from the number of errors that have occurred during decoding and can be expressed, for example, numerically. The match level (read margin) increases as error correction decreases and decreases as error correction increases.
[0075] A power cable (not shown) configured to supply power to the optical information reading device 1 from an external source is connected to the power connector 7. Additionally, signal lines 100a and 101a configured to communicate with the computer 100 and PLC 101 are connected to the signal line connector 8. The signal line connector 8 can be configured using, for example, an Ethernet connector, a serial communication connector such as RS-232C, or a USB connector.
[0076] The housing 2 is equipped with an indicator 9. The indicator 9 is connected to the processor 20 and can be configured using a light-emitting element such as a light-emitting diode. The operating status of the optical information reading device 1 can be notified externally by the illumination state of the indicator 9.
[0077] A pair of sight beam illuminators 10 are mounted on the front side of the housing 2 in a manner that clamps the camera 5. For example... Figure 3 As shown, the aiming light illumination unit 10 includes an aiming device 10a formed of a light-emitting diode or the like, and an aiming drive unit 10b for driving the aiming device 10a. The aiming device 10a emits light (aiming light) towards the front of the optical information reading device 1 to indicate the shooting range or field of view center of the camera 5, the guidance of the optical axis of the illumination unit 4, etc. Specifically, the aiming device 10a emits visible light of a different color (e.g., red or green) than ambient light towards the interior of the shooting field of view of the camera 5, and forms a mark visible to the naked eye on the surface irradiated with the visible light. The mark can be various graphics, symbols, or characters. The user can also refer to the light emitted from the aiming device 10a when installing the optical information reading device 1.
[0078] like Figure 2As shown, operation buttons 11 and 12 used in setting up the optical information reading device 1 are provided on the side of the housing 2. Operation buttons 11 and 12 include, for example, selection buttons and input buttons. In addition to operation buttons 11 and 12, a touch panel type operation mechanism may be provided, for example. Operation buttons 11 and 12 are connected to the processor 20, and the processor 20 can detect the operation status of operation buttons 11 and 12. One of several options displayed on the display unit 6 can be selected or the selected result can be confirmed by operating operation buttons 11 and 12.
[0079] (Handheld optical information reading device)
[0080] Although the example above describes the case where the optical information reading device 1 is a fixed type, this aspect can also be applied to optical information reading devices other than the fixed optical information reading device 1. Figure 4 A handheld optical information reading device 1A is shown, and this aspect can also be applied to, for example... Figure 4 The handheld optical information reading device 1A shown is shown.
[0081] The housing 2A of the handheld optical information reading device 1A has an elongated shape in the vertical direction. Note that the orientation of the optical information reading device 1A during use is not limited to the orientation shown, and the optical information reading device 1A can be used in various orientations, but the vertical direction of the optical information reading device 1A has been specified for ease of description.
[0082] The display unit 6A is disposed in the upper part of the housing 2A. The display unit 6A is configured similarly to the display unit 6 of the fixed optical information reading device 1. The lower part of the housing 2A is the grip part 2B, which is held by the user during operation. The grip part 2B is a part that can be held by an ordinary adult with one hand, and its shape and size can be freely set. The optical information reading device 1A can be carried and moved by holding the grip part 2B. That is to say, the optical information reading device 1A is a portable terminal device, and can also be referred to as a handheld terminal, for example.
[0083] The handheld housing 2A, like the fixed optical information reading device 1, also houses an illumination unit, a camera, a sight beam irradiation unit, a processor, a storage unit, a ROM, and RAM (not shown). The optical axes of the illumination unit, the camera, and the sight beam irradiation unit point obliquely upwards from near the upper end of the housing 2A. Additionally, the handheld housing 2A is equipped with a buzzer (not shown).
[0084] Multiple operation buttons 11A and trigger keys 11B are provided on and near the grip 2B. The operation buttons 11A are identical to the operation buttons 11 of the fixed optical information reading device 1. If the user presses the trigger key 11B while facing the workpiece W at the end (top) of the optical information reading device 1A, a sight light is emitted from the end of the optical information reading device 1A. If the user adjusts the direction of the optical information reading device 1A while visually observing the sight light reflected on the surface of the workpiece W and focusing the sight light on the code to be read, the code reading and decoding process is performed automatically. When reading is complete, a completion notification sound is generated from a buzzer.
[0085] As a use case of the handheld optical information reading device 1A, there are examples of using the handheld optical information reading device 1A for picking operations in a distribution warehouse. For example, in the case of delivering ordered products from a distribution warehouse, the necessary products are picked from product shelves in the product warehouse. This picking operation is performed while a user carrying an order ticket with a description code checks the code on the order ticket against the code attached to the product or product shelf after moving to the product shelf. In this case, the code on the order ticket and the code attached to the product or product shelf are read alternately by the handheld optical information reading device 1A.
[0086] (Processor configuration)
[0087] The following description is common to both the fixed optical information reading device 1 and the handheld optical information reading device 1A, and unless otherwise stated, it can also be applied to both devices 1 and 1A. Figure 3 As shown, processor 20 includes CPU core 21, DSP core 22, and AI chip 23. CPU core 21, DSP core 22, and AI chip 23 are a so-called system-on-a-chip (SoC, SOC) and are mounted on a substrate. Note that DSP core 22 and AI chip 23 do not necessarily have to be an SoC. In this case, DSP core 22 and AI chip 23 are not mounted on the same substrate, but this situation is also included within the scope of this aspect.
[0088] High-speed RAM 41 is connected to CPU core 21, DSP core 22, and AI chip 23, and all of CPU core 21, DSP core 22, and AI chip 23 can access RAM 41. Additionally, ROM 40 is connected to processor 20, and all of CPU core 21, DSP core 22, and AI chip 23 can access ROM 40.
[0089] CPU core 21 is a general-purpose processor and is used to perform functions such as AF control, lighting control, camera control, and decoding of coded images. DSP core 22 is used, for example, to perform various filtering processes on the coded images. AI chip 23 is an integrated circuit dedicated to attempting to recover coded images using neural networks, and is specifically designed to perform the product operations required by the neural network at ultra-high speed.
[0090] like Figure 5 As shown, the processor 20 is configured with an AF control unit 20a, an imaging control unit 20b, a filtering processing unit 20c, an adjustment execution unit 20d, a decoding processing unit 20f, an extraction unit 20g, a reduction unit 20h, a magnification unit 20i, and an image restoration unit 20j. The AF control unit 20a, imaging control unit 20b, filtering processing unit 20c, adjustment execution unit 20d, decoding processing unit 20f, extraction unit 20g, reduction unit 20h, and magnification unit 20i are configured through arithmetic processing by a CPU core 21 or a DSP core 22. On the other hand, the image restoration unit 20j is configured using an AI chip 23.
[0091] (Configuration of the AF control unit)
[0092] AF control unit 20a is used for control Figure 3 The AF module 5c shown is a unit that is configured to focus the optical system 5b using conventionally known contrast AF and phase difference AF.
[0093] (Configuration of the imaging control unit)
[0094] The imaging control unit 20b is a unit used to adjust the gain of the camera 5, control the amount of light from the illumination unit 4, and control the exposure time (shutter speed) of the imaging element 5a. Here, the gain of the camera 5 is the magnification factor (also called magnification) when the brightness of the image output from the imaging element 5a is amplified by digital image processing. The amount of light from the illumination unit 4 can be changed by controlling the first illumination unit 4a and the second illumination unit 4b separately. Gain, the amount of light from the illumination unit 4, and exposure time constitute the imaging conditions of the camera 5.
[0095] (Configuration of the filtering processing unit)
[0096] The filtering unit 20c is a section for performing image processing filtering on the image. The filtering unit 20c performs noise removal filtering to remove noise included in the image generated by the camera 5, contrast correction filtering to correct contrast, and averaging filtering, etc. The image processing filtering performed by the filtering unit 20c is not limited to noise removal filtering, contrast correction filtering, and averaging filtering, and may include other image processing filters.
[0097] The filtering unit 20c is configured to perform image processing filtering on the code image prior to image restoration, as described later. Additionally, the filtering unit 20c is configured to perform image processing filtering on the code image prior to magnification or reduction, as described later. Note that the filtering unit 20c can be configured to perform image processing filtering on the code image obtained after image restoration, or it can be configured to perform image processing filtering on the magnified or reduced code image.
[0098] (Adjusting the configuration of the execution unit)
[0099] Figure 5 The adjustment execution unit 20d shown repeats the imaging and decoding processes while changing the imaging conditions of the camera 5 and the decoding conditions of the decoding process. It determines the optimal imaging and decoding conditions based on the matching level of the ease of reading the indicator code calculated under each imaging and decoding condition, and performs adjustment processing to set the size of the code to be read. Specifically, the adjustment execution unit 20d is used to change the imaging conditions (such as the gain of the camera 5, the amount of light from the illumination unit 4, and the exposure time) and the image processing conditions in the filtering processing unit 20c when setting up the optical information reading devices 1 and 1A, and to set various conditions (adjustment parameters) to obtain conditions suitable for decoding. The image processing conditions in the filtering processing unit 20c include the coefficients of the image processing filter (the intensity of the filter), or the switching of image processing filters when multiple image processing filters exist, and combinations of different types of image processing filters. Appropriate imaging and image processing conditions vary depending on the influence of external light on the workpiece W during transport, and the color and material of the surface to which the code is attached. Therefore, the adjustment execution unit 20d searches for more appropriate imaging and image processing conditions, and sets up processing using the AF control unit 20a, the imaging control unit 20b, and the filter processing unit 20c.
[0100] The adjustment execution unit 20d is configured to acquire the size of the code included in the code image generated by the camera 5. When acquiring the code size, firstly, the adjustment execution unit 20d searches for a code based on feature quantities indicating code similarity. Next, the adjustment execution unit 20d acquires the pixel-per-unit (PPC), code type, and code size of the searched code. Since PPC, code type, and code size are included in code parameters or code conditions, the adjustment execution unit 20d is configured to acquire the code parameters or code conditions of the searched code.
[0101] The QR code is configured by randomly arranging multiple white and black modules. For example, in model 2, which serves as a QR code, the number of modules ranges from 21×21 to 177×177. In the case of a fixed optical information reading device 1, the code to be read is pre-read to limit the number of modules and PPC, and the operation is performed by limiting the size of the code (pixel size) to the number of modules × PPC.
[0102] When focusing on a module within a code-forming module, the code parameter indicating how many pixels form that module is is the PPC. The adjustment execution unit 20d can obtain the PPC by specifying a module and then counting the number of pixels forming that module.
[0103] The code type is, for example, a QR code, a data matrix code, or a verification code. Since each code has its own characteristics, the adjustment execution unit 20d can distinguish the code type based on, for example, the presence or absence of a lookup pattern.
[0104] Alternatively, the code size can be calculated using the PPC and the number of modules arranged vertically or horizontally in the code. The adjustment execution unit 20d calculates the number of modules arranged vertically or horizontally in the code, calculates the PPC, and combines the number of modules arranged vertically or horizontally with the PPC to obtain the code size.
[0105] (Configuration of the decoding processing unit)
[0106] The decoding processing unit 20f is used to decode the data that has been binarized into black and white. For decoding, a table indicating the comparison relationship of the encoded data can be used. Furthermore, the decoding processing unit 20f checks the correctness of the decoding result according to a predetermined checking scheme. When an error is found in the data, an error correction function is used to calculate the correct data. The error correction function varies depending on the code type.
[0107] In this embodiment, the decoding processing unit 20f decodes the codes included in the code image obtained after image restoration, which will be described later. However, it may also decode the codes included in the code image before image restoration. The decoding processing unit 20f is configured to write the decoding result obtained by decoding the codes into... Figure 3 The decoding result is stored in storage unit 30b of storage unit 30 shown.
[0108] (Configuration of the extraction unit)
[0109] The extraction unit 20g is used to extract code candidate regions with a high probability of code presence from the code image generated by the camera 5. Code candidate regions can be extracted based on features indicating code similarity. In this case, the features indicating code similarity are information used to identify the code. For example, the extraction unit 20g can acquire a code image and search for codes in the acquired code image based on the features indicating code similarity. Specifically, it searches for regions in the acquired code image that have predetermined or more features indicating code similarity. If a region with features indicating code similarity is found as a result, the region including that region is extracted as a code candidate region. The code candidate region can include regions other than codes, but at least includes regions with a predetermined or higher probability of being a code. Note that the code candidate region is only a region with a high probability of code presence; therefore, there may be cases where the code candidate region is the last region that does not include a code.
[0110] The extraction unit 20g can also extract a code candidate region by using the position information of the aiming light emitted from the aiming light irradiation unit 10 as information for identifying the code. In this case, the extraction unit 20g identifies the portion corresponding to the mark formed by the aiming light irradiation unit 10 from the code image and extracts the identified portion as a code candidate region. That is, since the aiming light indicates the shooting range and field of view center of the camera 5 as described above, the mark formed by the aiming light can be pre-set to coincide with the field of view center of the camera 5. In particular, in the case of the handheld optical information reading device 1A, the user focuses the aiming light on the code to be read, so the probability of the mark formed by the aiming light overlapping with the code is high. That is, when the extraction unit 20g extracts the region including the field of view center of the camera 5 by setting the portion corresponding to the mark formed by the aiming light irradiation unit 10 to coincide with the field of view center of the camera 5, this region is a region where the code is highly likely to exist. In this case, the portion corresponding to the mark formed by the aiming light irradiation unit 10 does not necessarily have to be set to coincide perfectly with the field of view center of the camera 5, and can be slightly shifted in the vertical or horizontal direction of the field of view. Since the code has a predetermined size, the area extracted by the extraction unit 20g is an area that not only has the center of the field of view of the camera 5, but also has a predetermined size including the center of the field of view.
[0111] The extraction unit 20g can also identify the central portion of the code image and extract this central portion as a code candidate region. For example, when the center of the field of view of the camera 5 is acquired in advance as information for code identification, the portion on the code image that is to become the center of the field of view of the camera 5 can be identified. In particular, in the case of the handheld optical information reading device 1A, the central portion identified on the code image corresponds to the mark formed by the aiming light irradiation unit 10, therefore, this central portion is an area where the code is highly likely to exist.
[0112] The extraction unit 20g can be configured to receive a user's designation of a predetermined portion from the code image and extract the designated portion as a code candidate region. That is, when a user designates a region of arbitrary size at any position (coordinates) on the code image, the extraction unit 20g receives the designation of the predetermined portion from the code image based on the coordinates and size information of that position. The extraction unit 20g extracts the received portion as a code candidate region. For example, in the case of a fixed optical information reading device 1, such as... Figure 1 As shown, a workpiece W conveyed by a conveyor belt B is photographed, and code decoding processing is performed. However, workpiece W does not necessarily have to be located in the center of the conveyor belt B in the width direction; there are cases where workpiece W is located at an end. In this case, by specifying the portion corresponding to the end of the conveyor belt B from the code image, the area where the code is highly likely to be present can be accurately extracted. Additionally, there are cases where the code appears near the end of a large workpiece W, away from the center. In this case, by specifying the portion corresponding to the vicinity of the end of the workpiece W from the code image, the area where the code is highly likely to be present can be accurately extracted.
[0113] (Storage configuration)
[0114] although Figure 3 The storage unit 30 shown can be configured using a readable / writable storage device such as a solid-state drive (SSD), but for example, the storage units 30a, 30b, 30c, and 30d, which will be described below, can each be located in the ROM 40 instead of a storage device. That is, this embodiment also applies the form in which the image data storage unit 30a, the decoding result storage unit 30b, the parameter set storage unit 30c, and the neural network storage unit 30d are also included in the ROM 40. The image data storage unit 30a stores the coded image generated by the camera 5. The decoding result storage unit 30b is the part used to store the decoding result of the code executed by the decoding processing unit 20f. The parameter set storage unit 30c is the part used to store the adjustment results performed by the adjustment execution unit 20d and various setting conditions, as well as various conditions set by the user. The neural network storage unit 30d is the part used to store the structure and parameters of the trained neural network, which will be described later.
[0115] (Image restoration using neural networks)
[0116] Optical information reading devices 1 and 1A have an image restoration function that uses a neural network to recover the code image acquired by camera 5. This embodiment differs from conventional optical information reading devices in that optical information reading device 1 or 1A pre-preserves the structure and parameters of a neural network that has already been learned (instead of performing machine learning through optical information reading device 1 or 1A), and uses the neural network configured with the preserved structure and parameters to perform image restoration.
[0117] like Figure 6 As shown, the neural network includes an input layer that takes in input data (image data in this example), an output layer that outputs output data, and intermediate layers positioned between the input and output layers. For example, multiple intermediate layers can be set to form a neural network with a multi-layer structure.
[0118] (Learning from neural networks)
[0119] First, based on Figure 7 The flowchart shown illustrates the basic process of neural network learning. Neural network learning can be performed using a computer specifically designed for learning, other than the optical information reading devices 1 and 1A; however, it can also be performed using a general-purpose computer other than the computer specifically designed for learning. Furthermore, conventionally known methods can be used as neural network learning methods.
[0120] In step SA1 after the start, the data of the pre-prepared defect image is read. The defect image is an image with portions that are unsuitable for reading codes, and can be read through methods such as... Figure 8A The image shown is used to illustrate this. Unsuitable parts are, for example, dirty or colored parts. Afterwards, processing proceeds to step SA2 to read the data of the pre-prepared ideal image. The ideal image is an image suitable for reading codes and can be obtained through methods such as... Figure 8B The image shown is used as an example. Defective images are paired with ideal images, and multiple such pairs are prepared.
[0121] In step SA2, a loss function is calculated to obtain the difference between the defective image and the ideal image. In step SA3, the parameters of the neural network are updated by reflecting the difference obtained in step SA2.
[0122] In step SA4, it is determined whether the completion conditions for machine learning are met. The completion conditions can be set based on the difference obtained in step SA2, and for example, if the difference is equal to or less than a predetermined value, it can be determined that the completion conditions for machine learning are met. If step SA4 determines "yes" and the completion conditions for machine learning are met, the machine learning process ends. On the other hand, if step SA4 determines "no" and the completion conditions for machine learning are not met, the process returns to step SA1 to read another defective image, and then proceeds to step SA2 to read another ideal image paired with that other defective image.
[0123] In this way, a trained neural network can be pre-generated by performing machine learning on multiple defective images and multiple ideal images corresponding to these defective images. The trained neural network can be constructed within optical information reading devices 1 and 1A while maintaining the structure and parameters of the trained neural network. That is, during machine learning, a large number of defective and ideal images are input into the neural network, thus requiring extremely high computational power. It is difficult to ensure such high computational power within optical information reading devices 1 or 1A because it may be difficult to meet the requirements for reducing the size and weight of the device. As in this example, if machine learning is performed outside optical information reading devices 1 or 1A, and the resulting neural network structure and parameters are maintained within optical information reading devices 1 or 1A, the image can be recovered using the neural network while reducing the size and weight of optical information reading devices 1 or 1A.
[0124] Figure 9 This is a conceptual diagram of a Convolutional Neural Network (CNN) configured as described above. In "convolution," features are extracted from the input image. This layer is configured through convolutional operations such as image processing filtering. The weights of the filter are called kernels, and the amount of features extracted is based on the kernel. Convolutional layers typically have multiple different kernels, and the number of mappings (D) increases with the number of kernels. The kernel size can be set to, for example, 3×3 or 5×5, etc.
[0125] In "Pooling," a scaling process is performed to collect the responses of each kernel. In "Deconvolution," the image is reconstructed based on the feature values through deconvolution filtering. In "Depooling," a scaling process is performed to sharpen the response values.
[0126] The structure and parameters of the trained neural network are stored in Figure 3 The storage unit 30 shown is located in the neural network storage unit 30d. The structure of the neural network includes the number (D) of intermediate layers between the input layer and the output layer, as well as the number of image processing filters, etc. The parameters of the neural network are... Figure 7 The parameters set in step SA3 of the flowchart shown.
[0127] Furthermore, the neural network storage unit 30d can store the structures and parameters of multiple neural networks with different numbers of layers or different numbers of image processing filters. In this case, the structures and parameters of multiple neural networks, along with code conditions, can be stored in the neural network storage unit 30d in association with each other. For example, the structure and parameters of the neural network used to form the first neural network are associated with a first code condition corresponding to the first neural network. Since the number of layers or the number of filters in the first neural network can be pre-specified by the first code condition (especially the first PPC) when the association is performed, this association is performed to maintain this relationship. Similarly, the structure and parameters of a second neural network different from the first neural network are associated with a second code condition different from the first code condition.
[0128] When the structures and parameters of multiple neural networks are stored in the neural network storage unit 30d, the adjustment execution unit 20d operates as follows when setting the optical information reading device 1 or 1A. That is, when setting the code conditions included in the code image generated by the camera 5, the adjustment execution unit 20d reads the structure and parameters of the neural network associated with the set code conditions from the neural network storage unit 30d, and identifies the read structure and parameters as the neural network to be used during operation. For example, when the adjustment execution unit 20d sets the first code condition by setting the code conditions included in the code image, it reads the structure and parameters of the first neural network associated with the first code condition from the neural network storage unit 30d. Then, the structure and parameters of the first neural network are identified as the neural network to be used during the operation of the optical information reading device 1 or 1A. As a result, the code image can be recovered using the structure and parameters of the neural network optimal for the code conditions, thus improving reading accuracy.
[0129] In neural network learning, machine learning can also be performed on defective images where the pixel resolution of the ideal image and the module forming the code falls within a specific range. The pixel resolution of the module can be represented by, for example, the number of pixels (PPC) of a module forming the code, and machine learning of the neural network can be performed using only the ideal image and defective images where the PPC falls within a specific range.
[0130] The specific range includes pixel resolutions that achieve high restoration results for code images and excludes pixel resolutions where the restoration results for code images are hardly improved. This range can be set while still allowing the restoration of images containing codes formed by multiple modules. As a result, a neural network structure specifically designed for restoring images containing codes can improve processing speed.
[0131] The specific range of pixel count can be set, for example, to be between 4 PPC and 6 PPC, but it can also be less than 4 PPC or more than 6 PPC. Additionally, for example, when machine learning is performed within a specific range limited to between 4 PPC and 6 PPC, additional scale variations can be eliminated from the images used for learning, and the trained neural network structure can be optimized.
[0132] (Recovery Attempt)
[0133] like Figure 5 As shown, the processors 20 of both optical information reading devices 1 and 1A are equipped with image restoration units 20j configured using an AI chip 23. The image restoration unit 20j reads the structure and parameters of a neural network stored in the neural network storage unit 30d of the storage unit 30, and uses the read neural network structure and parameters to configure the neural network in optical information reading device 1 or 1A. The neural network configured in optical information reading device 1 or 1A is compared with the neural network structure and parameters already configured using an AI chip 23. Figure 7 The process shown in the flowchart is the same as that of the trained neural network that learns.
[0134] The image restoration unit 20j inputs the coded image generated by the camera 5 into the neural network and attempts to restore the coded image based on the structure and parameters of the neural network stored in the neural network storage unit 30d. Then, the decoding processing unit 20f performs decoding processing on the restored coded image.
[0135] The filtering unit 20c is configured to perform image processing filtering on the code image before the image restoration unit 20j inputs the code image generated by the camera 5 into the neural network. As a result, appropriate image processing can be performed before attempting restoration via the neural network. Consequently, the restoration of the code image becomes more accurate.
[0136] The image restoration unit 20j may attempt to restore all the code images processed by the decoding processing unit 20f, or it may attempt to restore only a portion of the code images processed by the decoding processing unit 20f. For example, the decoding processing unit 20f performs decoding processing on the code image before attempting restoration and determines whether the decoding was successful. Successful decoding means that the code image does not need to be restored, and the result is output directly. On the other hand, if decoding fails, the image restoration unit 20j attempts to restore the code image, and the decoding processing unit 20f performs decoding processing on the restored code image.
[0137] (Input of a portion of the image)
[0138] The image input to the neural network by the image restoration unit 20j can be a whole-image generated by the camera 5. However, as the size of the input image increases, the computational load on the neural network increases, and the computational load on the processor 20 becomes enormous, potentially leading to a decrease in processing speed. Specifically, in cases such as Figure 9 In the fully convolutional network (FCN) type neural network shown, when assuming equal structures, the computational cost is proportional to the size of the input image, and the size of the input image is proportional to the square of the code size. Therefore, the computational cost is also proportional to the square of the code size.
[0139] On the other hand, it can be said that the following situation does not exist: the code exists in the entire code image generated by camera 5. During normal operation, the code only exists in a portion of the code image, and if recovery using a neural network can only be performed in that portion, the reading accuracy can be improved.
[0140] Therefore, the image input to the neural network by the image restoration unit 20j can be a portion of the code image generated by the camera 5, that is, a partial image including the code. Specifically, the image restoration unit 20j cuts out a partial image corresponding to the code candidate region extracted by the extraction unit 20g from the code image generated by the camera 5. The image restoration unit 20j inputs the cut partial image into the neural network and attempts to restore the partial image based on the structure and parameters stored in the neural network storage unit 30d. Then, the decoding processing unit 20f performs decoding processing on the restored partial image.
[0141] As described above, since it is almost impossible for a code to exist in the entire code image, the size of the partial image corresponding to the code candidate region is smaller than the size of the code image. As a result, the size of the image input to the neural network is reduced, and therefore, the computational load on the processor 20 is reduced, leading to an increase in processing speed. Furthermore, since the partial image corresponds to the region where the code is highly likely to exist, the partial image can be an image that includes the information needed for reading (i.e., the entire code). Because an image including the entire code is input to the neural network, a decrease in reading accuracy is suppressed.
[0142] The image restoration unit 20j can determine the input size of the partial image to the neural network based on the code size obtained by the adjustment execution unit 20d. As described above, the adjustment execution unit 20d can obtain the code size. The image restoration unit 20j increases the input size of the partial image to the neural network by a predetermined amount from the code size obtained by the adjustment execution unit 20d.
[0143] In other words, the computational load can be reduced by inputting a portion of the image into the neural network. However, for example, in cases where the code is rotating or the code's position cannot be accurately detected, there is a possibility that a portion of the code in the partial image may be lost, leading to decoding failure. In this embodiment, since the input size to the neural network is different from the code size obtained by the adjustment execution unit 20d and is larger than the code size by a predetermined amount, the loss of a portion of the code in the partial image can be suppressed. Note that by making the input size larger than the code size by a predetermined amount, the input size to the neural network can have an upper limit, thus preventing an increase in computational load. Regarding the "predetermined amount," for example, the horizontal (or vertical) length of the partial image input to the neural network can be set to 1.5 times or more, 2.0 times or more, or 3.0 times or more of the horizontal (or vertical) length of the code obtained by the adjustment execution unit 20d.
[0144] When the image restoration unit 20j determines the input size of the portion of the image to be input into the neural network, the input size can be determined based on the number of pixels of a module forming a code acquired by the adjustment execution unit 20d and the number of modules arranged in the vertical or horizontal direction of the code.
[0145] The adjustment execution unit 20d can set the code conditions as described above. The image restoration unit 20j can also set the size of the code image to be input into the neural network based on the code conditions set by the adjustment execution unit 20d. For example, if the image restoration unit 20j acquires the code size in the code conditions set by the adjustment execution unit 20d, it inputs a portion of the image that is larger than the acquired code size into the neural network. When the code size is large, the size of the portion of the image input into the neural network increases, and when the code size is small, the size of the portion of the image input into the neural network decreases. In other words, the size of the portion of the image input into the neural network can be changed according to the code conditions.
[0146] (Zoom out and zoom in function)
[0147] Here, it is necessary to extract feature quantities within a range covering a certain number of modules, in order to achieve, for example Figure 9 The convolutional neural network shown fully captures the features of the code and recovers the code into an image suitable for decoding. For example, assuming a feature value obtained from the neural network is extracted from a range of 6×6 modules, then in the case of an image captured by a module with 20 pixels, the neural network needs to be designed to compute aggregated feature values from a range of 120×120 pixels. That is, as the desired pixel range becomes wider, the layers of the neural network need to be deeper. For example, when aggregating feature values from the aforementioned 120×120 pixel range, six convolutional layers need to be applied.
[0148] On the other hand, by configuring the code by randomly arranging white and black modules, the relationship between pixel values over a wide range is sparse, and even if the range of pixels to be covered is expanded to a certain extent or larger, the restoration effect is hardly improved. Such a feature is referred to as the narrow-range feature in this specification.
[0149] Furthermore, the arrangement of the determined module patterns (such as the finder pattern included in the code or the quadrilateral shape of the entire code) can be a wide-range feature. However, as long as the modules and background are roughly separated, reconstruction can be sufficiently performed using only narrow-range features without using wide-range features. Note that a wide-range feature can be defined as a feature with a shape determined over a wide range.
[0150] Therefore, in Figure 7 The neural network shown can learn by using images within a specific range of the code's PPC. As a result, processing speed can be improved because the neural network structure is specifically designed to recover images including the code; however, there is a concern about the degradation in recovery performance for code images where the PPC is outside the specific range. Note that images outside the specific range of the code's PPC can be used during neural network learning.
[0151] The optical information reading devices 1 and 1A of this embodiment have the function of reducing and enlarging the code image so that the PPC of the code image falls within a specific range. For example... Figure 5 As shown, the processor 20 includes a reduction unit 20h and an enlargement unit 20i. The reduction unit 20h is the portion used to generate a reduced code image such that the pixel resolution of the module forming the code in the code image generated by the camera 5 falls within the aforementioned specific range. Similarly, the enlargement unit 20i is the portion used to generate an enlarged code image such that the pixel resolution of the module forming the code in the code image generated by the camera 5 falls within the aforementioned specific range.
[0152] For example, the reduction unit 20h and the magnification unit 20i determine whether the PPC of the code in the code image generated by the camera 5 is outside a specific range (4 PPC or more and 6 PPC or less). If the PPC of the code falls within the specific range, no reduction or magnification is performed, and if the PPC of the code is outside the specific range, reduction or magnification is performed. When reduction or magnification is performed, the PPC of the code in the portion of the image input to the neural network by the image restoration unit 20j falls within the specific range.
[0153] The image restoration unit 20j inputs a code image that has been magnified or reduced to fall within a specific range into a neural network configured using the structure and parameters stored in the neural network storage unit 30d, thereby attempting to restore the code image based on the structure and parameters. Then, the decoding processing unit 20f performs decoding processing on the restored code image.
[0154] The extraction unit 20g can be configured to search for codes with respect to a code image magnified by the magnification unit 20i or a code image reduced by the reduction unit 20h, and extract regions including the searched codes as code candidate regions with a high probability of code existence. In this case, the image restoration unit 20j inputs a portion of the image corresponding to the code candidate region extracted by the extraction unit 20g into a neural network configured using the structure and parameters stored in the neural network storage unit 30d, thereby attempting to restore the code image based on the structure and parameters.
[0155] When the optical information reading device 1 or 1A is set up, the adjustment execution unit 20d can generate multiple code images (magnified images) with different magnification ratios and multiple code images (reduced images) with different reduction ratios. The adjustment execution unit 20d inputs the generated multiple code images (magnified or reduced images) into a neural network to attempt to recover each code image, and performs decoding processing on each recovered code image to obtain a read margin indicating the ease of code reading. The adjustment execution unit 20d identifies the magnification ratio or reduction ratio of code images whose obtained read margin is higher than a predetermined value as the magnification ratio or reduction ratio to be used during operation. When the adjustment execution unit 20d has identified the magnification ratio or reduction ratio, the magnification unit 20i and the reduction unit 20h magnify or reduce the code image at the magnification ratio or reduction ratio identified by the adjustment execution unit 20d during the operation of the optical information reading devices 1 and 1A.
[0156] In other words, for example, when a certain range has a certain degree of limitation, it is conceivable that if the magnification or reduction rate changes even within the specific range, the read margin also changes. Since the magnification or reduction rate of the code image during operation can be identified, the read margin obtained by the adjustment execution unit 20d becomes higher than the predetermined value, thereby improving processing speed and read accuracy.
[0157] After obtaining the read margin of each code image, the adjustment execution unit 20d can identify the magnification or reduction rate of the code image with the highest read margin among the obtained multiple read margins as the magnification or reduction rate to be used during operation. As a result, processing speed and read accuracy can be further improved.
[0158] Furthermore, the types of parameter sets can be increased by utilizing the fact that the code image can be reduced and enlarged by the reduction section 20h and the enlargement section 20i. Multiple parameter sets with different reduction rates of the reduction section 20h and multiple parameter sets with different magnification rates of the enlargement section 20i are generated, and these parameter sets can be stored in the parameter set storage section 30c of the storage section 30. For example, if the reduction section 20h can be set with multiple reduction rates (such as 1 / 2, 1 / 4, and 1 / 8), parameter sets with a reduction rate of 1 / 2, 1 / 4, and 1 / 8 can be stored in the parameter set storage section 30c. Similarly, if the enlargement section 20i can be set with multiple magnification rates (such as 2x, 4x, and 8x), parameter sets with a magnification rate of 2x, 4x, and 8x can be stored in the parameter set storage section 30c. Then, any parameter set from the multiple parameter sets stored in the parameter set storage section 30c can be applied.
[0159] (Image restoration filtering)
[0160] The image restoration unit 20j may be a part used to perform image restoration filtering attempts using a neural network to recover the code image. Image restoration filtering is a filtering process that attempts to recover the code image by inputting the code image into a neural network including the structure and parameters stored in the neural network storage unit 30d, based on the structure and parameters stored in the neural network storage unit 30d, and exhibits the same function as the image restoration described above.
[0161] When image restoration filtering can be performed, a setting unit 20e for setting the image restoration filtering can be provided. Figure 5 (As shown). The setting unit 20e is configured to receive settings for image restoration filtering made by the user. For example, when setting up the optical information reading device 1 or 1A, the setting unit 20e can be configured to generate a user interface that allows the user to select either "apply image restoration filtering" or "not apply image restoration filtering," display this user interface on the display unit 6, and receive the user's selection. Therefore, the image restoration unit 20j applies the image restoration filtering set by the setting unit 20e to the code image in an attempt to restore the code image.
[0162] The setting unit 20e may be included in the adjustment execution unit 20d, or it may be configured separately from the adjustment execution unit 20d. When the setting unit 20e is included in the adjustment execution unit 20d, parameters related to image restoration filtering can be set during adjustment. The parameters related to image restoration filtering may include "applying image restoration filtering" and "not applying image restoration filtering." "Applying image restoration filtering" means that image restoration filtering is set to be executable, and "not applying image restoration filtering" means that image restoration filtering is not set.
[0163] The parameters related to image restoration filtering are also information related to image restoration filtering. In this case, a parameter set including information related to the setting of image restoration filtering can be stored in the parameter set storage unit 30c. Since the parameters related to image restoration filtering include "application of image restoration filtering" and "non-application of image restoration filtering", the parameter set stored in the parameter set storage unit 30c includes a first parameter set for setting image restoration filtering to be executable and a second parameter set for not setting image restoration filtering. During the operation of the optical information reading device 1 or 1A, a parameter set selected from the first parameter set and the second parameter set is applied. When the first parameter set is applied, the decoding processing unit 20f performs decoding processing on the code image that has already undergone image restoration filtering. On the other hand, when the second parameter set is applied, the decoding processing unit 20f performs decoding processing on the code image that has not yet undergone image restoration.
[0164] The parameter set stored in the parameter set storage unit 30c also includes items for setting code conditions included in the code image generated by the camera 5. Among the items for setting code conditions are PPC, code type, code size, etc., acquired by the adjustment execution unit 20d. For example, a parameter set may include: PPC, code type, and code size as items for setting code conditions; the gain of the camera 5 as imaging conditions; the light amount and exposure time of the illumination unit 4; the image processing filter type as an item for image processing filtering applied by the filter processing unit 20c; the parameters of the image processing filter; and the application item for image restoration filtering. The values set by the adjustment execution unit 20d can be directly used as these items, or the user can arbitrarily change these items.
[0165] (Example of the adjustment process)
[0166] Reference Figure 10 The flowchart shown illustrates an example of the adjustment steps performed by the adjustment execution unit 20d when setting up the optical information reading device 1 or 1A. Figure 10 In step SB1, following the start of the flowchart shown, the execution unit 20d controls the illumination unit 4 and the camera 5 to cause the camera 5 to generate a coded image, and the execution unit 20d acquires the coded image. At this time, the presence and type of image processing filtering performed before decoding, as well as the decoding processing parameters for applying image restoration using a neural network, are set to arbitrary parameters. Next, the process proceeds to step SB2, in which the execution unit 20d causes the decoding processing unit 20f to perform decoding processing on the acquired coded image.
[0167] After decoding, the process proceeds to step SB3, and the execution unit 20d determines whether the decoding process in step SB2 was successful. If step SB3 determines "no" and the decoding process in step SB2 failed (i.e., code reading failed), the process proceeds to step SB4, where the decoding parameters are changed to another parameter, and then the decoding process is executed again in step SB2. If decoding using all decoding parameters fails, the process ends, and a notification is sent to the user.
[0168] On the other hand, if the determination in step SB3 is "yes" and the decoding process in step SB2 has been successful, the process proceeds to step SB5, and the execution unit 20d is adjusted to determine the code parameters (PPC, code type, code size, etc.). When the execution unit 20d determines the code parameters, the structure and parameters of the neural network stored in the neural network storage unit 30d associated with the code parameters are also determined (step SB6). In step SB6, the neural network is configured using the structure and parameters read from the neural network storage unit 30d.
[0169] In step SB7, the image restoration unit 20j attempts to restore the code image by inputting the code image into the neural network configured in step SB6, and the decoding processing unit 20f performs decoding processing on the restored code image. Thereafter, the process proceeds to step SB8, and the adjustment execution unit 20d evaluates the read margin based on the result of the decoding processing in step SB7, and temporarily stores the evaluation result.
[0170] In step SB9, it is determined whether the decoding process has been completed using all decoding parameters. If the determination in step SB9 is "no" and the decoding process has not been completed using all decoding parameters, the process proceeds to step SB10, where the decoding parameters are changed to another parameter, and then the process proceeds to step SB7.
[0171] On the other hand, if the determination in step SB9 is "yes" and the decoding process has been completed using all decoding processing parameters, the process proceeds to step SB11. In step SB11, the adjustment execution unit 20d selects the decoding processing parameter with the highest read margin from all decoding processing parameters and determines the selected decoding processing parameter as the parameter to be applied during operation. Note that in the adjustment step, imaging conditions, etc., are also set to appropriate conditions.
[0172] (Decoding process before determining the reduction and magnification rates)
[0173] Next, we will refer to Figure 11 The flowchart shown illustrates an example of the decoding process before determining the reduction and magnification rates. (It can be found...) Figure 10 Step SB2 of the flowchart shown is executed. Figure 11 The process specified in the flowchart shown.
[0174] exist Figure 11 In step SC1, following the start of the flowchart shown, the adjustment execution unit 20d selects an arbitrary image processing filter from a plurality of image processing filters. This image processing filter is the one to be performed by the filtering processing unit 20c. Subsequently, in step SC2, the filtering processing unit 20c performs the image processing filter selected in step SC1 on the code image.
[0175] Next, the process proceeds to step SC3 to create an image pyramid. The image pyramid includes the original code image, an image obtained by reducing the original code image to half its original size, an image obtained by reducing the original code image to one-quarter its original size, an image obtained by reducing the original code image to one-eighth its original size, and so on. The reduction of the code image is performed by the reduction unit 20h. Alternatively, the image pyramid may also include the original code image, an image obtained by enlarging the original code image by two times its original size, an image obtained by enlarging the original code image by four times its original size, an image obtained by enlarging the original code image by eight times its original size, and so on. The enlargement of the code image is performed by the enlargement unit 20i. Note that either the reduced image or the enlarged image can be omitted.
[0176] After creating the image pyramid, the process proceeds to step SC4, where an arbitrary code image is selected from the multiple code images that form the image pyramid. There is also a possibility that the selected image includes the original code image that has not been scaled down or enlarged. In step SC5, the extraction unit 20g extracts code candidate regions with a high probability of code presence from the code image selected in step SC4. Subsequently, the process proceeds to step SC6, where the image restoration unit 20j inputs a portion of the image corresponding to the code candidate regions extracted in step SC5 into the neural network to attempt to restore the code image. The size of the portion of the image input into the neural network is determined by the aforementioned code conditions.
[0177] After recovering the code image, the process proceeds to step SC7 to perform processing for locating the code outline and forming the code modules. After the location processing, the process proceeds to step SC8 to distinguish whether each module forming the code is white or black. After distinguishing between black and white, the process proceeds to step SC9, and the decoding processing unit 20f performs decoding processing on the recovered code image. In the decoding processing, for example, techniques for recovering strings from a (0,1) matrix of modules can be used.
[0178] Subsequently, the process proceeds to step SC10, and the execution unit 20d determines whether the decoding process in step SC9 has been successful. If the determination in step SC10 is "No" and the decoding process in step SC9 has failed, the process proceeds to step SC11 to determine whether all code images forming the image pyramid have been selected. If the determination in step SC11 is "No" and all code images forming the image pyramid have not been selected, the process proceeds to step SC4 to select another scaled-down code image or another scaled-up code image forming the image pyramid, and the process proceeds to step SC5.
[0179] On the other hand, if the determination in step SC11 is "yes" and all code images that form the image pyramid have been selected, the process ends and the condition that decoding has been successful is stored.
[0180] (Decoding process after determining the reduction and magnification rates)
[0181] Next, we will refer to Figure 12 The flowchart shown illustrates an example of the decoding process after the reduction and magnification rates are determined. (It can be found...) Figure 10 Step SB7 of the flowchart shown is performed, as well as during the operation of optical information reading device 1 or 1A. Figure 12 The process specified in the flowchart shown.
[0182] exist Figure 12 In step SD1 after the start of the flowchart shown, the filtering processing unit 20c performs the following steps on the code image: Figure 11 The image processing filter selected in step SC1 of the flowchart shown. At this time, when the filtering processing unit 20c performs image processing on the image... Figure 13A When the code image shown is subjected to averaging filtering, the following is obtained: Figure 13B The image shown is the result of image processing filtering.
[0183] Subsequently, the process proceeds to step SD2 to perform reduction or enlargement processing on the code image as needed. Specifically, if the PPC of the code in the code image obtained after image processing filtering falls within a specific range, no reduction or enlargement is performed; conversely, if the PPC is outside the specific range, reduction or enlargement is performed to bring the PPC within the specific range. Figure 14A The image shown is an example of a code image obtained after the downsizing process.
[0184] Next, in step SD3, the extraction unit 20g extracts code candidate regions with a high probability of code presence from the code image that was scaled down or enlarged in step SD2. Figure 14BThe image shown is an example of code candidate region extraction. Note that the extraction process is performed on the original code image in step SD3 if the original code image is not scaled down or scaled up in step SD2.
[0185] Subsequently, the process proceeds to step SD4, and the image restoration unit 20j inputs a portion of the image corresponding to the code candidate region extracted in step SD3 into the neural network to attempt to restore the code image. Figure 15 Examples of the code image before and after recovery are shown. After the code image is recovered, the process ends after steps SD5 to SD7. Steps SD5 to SD7 are... Figure 11 Steps SC7 to SC9 in the flowchart shown are the same.
[0186] (Functions and effects of the embodiments)
[0187] According to this embodiment, during the operation of the optical information reading device 1 or 1A as described above, when the code image generated by the camera 5 is input into the neural network, an attempt is made to recover the code image. Therefore, a code image obtained after the recovery attempt can be obtained, which has been recovered into an image suitable for code reading. Since the code image obtained after the recovery attempt is decoded, the reading accuracy is improved.
[0188] Because the neural network used for recovery is configured with structures and parameters obtained through machine learning from multiple defective images and multiple ideal images in advance, the optical information reading device 1 or 1A only needs to perform inference processing through the neural network without learning processing, thereby reducing the computational load on the processor 20. As a result, the size and weight of the handheld optical information reading device 1A can be reduced, and the processing speed of the fixed optical information reading device 1 can be improved.
[0189] Furthermore, when extracting code candidate regions from the code image during the operation of the optical information reading device 1 or 1A, a portion of the image corresponding to the code candidate region is input into the neural network to attempt to recover the portion of the image. As a result, the size of the image input to the neural network is reduced, and therefore, the computational load on the processor 20 is reduced, leading to an increase in processing speed. Additionally, since the portion of the image corresponds to the region where the code is highly likely to exist, the portion of the image can be an image including the information required for reading, i.e., the entire code. Because an image including the entire code is input into the neural network, a decrease in reading accuracy is suppressed.
[0190] Furthermore, machine learning is performed on both ideal images and defective images where the pixel resolution of the code-forming modules falls within a specific range when generating the structure and parameters of the neural network. Therefore, since the neural network structure is specifically designed to recover images including codes, processing speed can be improved. When the pixel resolution of the code-forming modules in the code-forming image is outside the specific range, the code image is enlarged or reduced to fall within that range before being input into the neural network. Thus, images including codes with a wide range of pixel resolutions can be recovered.
[0191] (Other embodiments)
[0192] Figure 16 This diagram illustrates an example of the processing procedure during operation of the optical information reading device 1 according to another embodiment. In this embodiment, apart from the image restoration unit 20j, the processor 20 is configured with a preprocessing unit 200, a post-processing unit 201, a first decoding processing unit 202, and a second decoding processing unit 203. The preprocessing unit 200 is the same as the filtering processing unit 20c, used to perform noise removal filtering, contrast correction filtering, and averaging filtering on the coded image generated by the camera 5.
[0193] The first decoding processing unit 202 is a unit used to perform decoding processing on the coded image generated by the camera 5, and the decoding processing performed by the first decoding processing unit 202 is also referred to as the first decoding processing. In the case where the processor 20 has multiple cores (N cores), the first decoding processing unit 202 is composed of cores 0 to 1. The first decoding processing unit 202 may include one core.
[0194] Furthermore, the second decoding processing unit 203 is used to perform decoding processing on the recovered image generated by the image restoration unit 20j, and the decoding processing performed by the second decoding processing unit 203 is also referred to as the second decoding processing. When the processor 20 has N cores, the second decoding processing unit 203 is composed of cores i+1 to N-1. The second decoding processing unit 203 may include one core.
[0195] For example Figure 17 As shown, when the first decoding processing unit 202 is composed of kernels 0 to i, each of kernels 0 to i performs code contour and code forming module positioning processing after extracting code candidate regions with a high probability of code presence from the code image generated by the camera 5. Then, it distinguishes whether each code forming module is white or black and performs decoding processing. The processing result is output to the post-processing unit 201.
[0196] On the other hand, such as Figure 17As shown, when the second decoding processing unit 203 is composed of kernels i+1 to N-1, each of kernels i+1 to N-1 extracts a code candidate region with a high probability of code presence from the code image as a code region for recovery, and sends a trigger signal to the image recovery unit 20j to perform inference processing. Although Figure 17 This shows that a task is executed by two kernels, but a task can be executed by one or more arbitrary kernels.
[0197] Specifically, when kernels N-2 and N-1, which form the second decoding processing unit 203, extract code candidate regions for recovery, a trigger signal (trigger 1) is output to the image recovery unit 20j to cause the image recovery unit 20j to perform inference processing. When the trigger signal (trigger 1) sent from kernels N-2 and N-1 is received, the image recovery unit 20j inputs a portion of the image corresponding to the code candidate regions extracted by kernels N-2 and N-1 for recovery into the neural network, and performs inference processing (image recovery 1) to generate a recovered image of the recovered portion of the image. The recovered image generated in image recovery 1 is sent to kernels N-2 and N-1. Kernels N-2 and N-1 determine the grid positions of each unit position of the indicator code based on the recovered image generated in image recovery 1, and perform decoding processing of the recovered image based on the determined grid positions.
[0198] On the other hand, when kernels i+1 and i+2, which form the second decoding processing unit 203, extract code candidate regions for recovery, a trigger signal (trigger 2) is output to the image recovery unit 20j to cause the image recovery unit 20j to perform inference processing. When the trigger signal (trigger 2) sent from kernels i+1 and i+2 is received, the image recovery unit 20j inputs a portion of the image corresponding to the code candidate regions for recovery extracted by kernels i+1 and i+2 into the neural network, and performs inference processing (image recovery 2) to generate a recovered image of the recovered portion of the image. The recovered image generated in image recovery 2 is sent to kernels i+1 and i+2. Kernels i+1 and i+2 determine the grid positions of each cell position of the indicator code based on the recovered image generated in image recovery 2, and perform decoding processing of the recovered image based on the determined grid positions.
[0199] In the following text, similarly, image restoration 3 is executed by trigger signals (trigger 3) sent from kernel N-2 and kernel N-1, image restoration 4 is executed by trigger signals (trigger 4) sent from kernel i+1 and kernel i+2, image restoration 5 is executed by trigger signals (trigger 5) sent from kernel N-2 and kernel N-1, image restoration 6 is executed by trigger signals (trigger 6) sent from kernel i+1 and kernel i+2, and image restoration 7 is executed by trigger signals (trigger 7) sent from kernel N-2 and kernel N-1.
[0200] As shown in the timing diagram, the inference process using the image restoration unit 20j and the first decoding process using the first decoding processing unit 202 are executed in parallel. For example, while the first decoding processing unit 202 continuously performs code region extraction, grid position determination, and decoding processes, the image restoration unit 20j performs the inference process once or multiple times. Furthermore, while the first decoding processing unit 202 continuously performs code region extraction, grid position determination, and decoding processes, the second decoding processing unit 203 performs the extraction of the code region to be restored, the output of the trigger signal, the determination of the grid position, and the second decoding process. Since the image restoration unit 20j only needs to be dedicated to restoring the portion of the image corresponding to the code candidate region to be restored in this manner, the speed of the inference process can be improved.
[0201] At least a portion of the time period during which the second decoding processing unit performs the second decoding processing has a pause period, during which the inference processing using the image restoration unit 20j is not performed. That is, as Figure 17 As shown, the second decoding processing unit 203 begins the second decoding process after the image restoration 1 of the image restoration unit 20j ends, but provides a predetermined period of time until the image restoration unit 20j begins the next image restoration 2, and this predetermined period of time is a pause period during which the image restoration unit 20j does not perform image restoration. This pause period is a period during which the image restoration unit 20j does not perform inference processing, and by providing this pause period, the load on the image restoration unit 20j can be reduced and the heat generation of the AI chip 23 can be suppressed. Note that the processing time for determining the grid position can be long or short depending on the image to be restored. Although Figure 17 The example shown illustrates the alternating execution of processes from individual kernels; however, based on the processing time determined by the grid position, it is not necessary to alternate the execution of processes from individual kernels. In other words, trigger emission and grid position determination can be performed sequentially, starting from the kernels available for processing.
[0202] As described above, the setting unit 20e is configured to allow the user to select either "applying image restoration filtering" or "not applying image restoration filtering." Applying image restoration filtering means performing image restoration. Conversely, not applying image restoration filtering means not performing image restoration. Therefore, the setting unit 20e also corresponds to the part used to set whether the second decoding processing unit 203 performs the second decoding process. Since this setting is performed by the user, the setting unit 20e is configured to receive user operations. The setting form is not limited to applying or not applying image restoration filtering, and for example, setting forms such as applying or not applying inference processing (restoration processing) and applying or not applying processing performed using AI can be used. Examples of user operations include operations on buttons displayed on the user interface screen.
[0203] After adjusting the imaging conditions, such as exposure time and gain, and the settings, such as image processing filtering, are predetermined, the optical information reading device 1 is operated. For example, in the case where the workpiece W is a label, the code is clearly printed, so a high reading rate can be achieved if appropriate conditions are set through the adjustment process. However, for workpiece W made of metal or resin, even if the optimal conditions are set during the adjustment process, there may be situations where reading becomes difficult due to changes in the code image caused by weak ambient light, or where reading with extremely low contrast is difficult due to the original optical and algorithmic design.
[0204] When the image restoration algorithm executed by the image restoration unit 20j in this example is performed in parallel with decoding processing without image restoration, the reading performance of workpieces W made of metal or resin can be improved while maintaining conventional reading performance and enhancing reading stability. Furthermore, the image restoration algorithm can be applied to workpieces W that are unreadable by conventional optical designs and algorithms.
[0205] As a concrete example, in the process of reading the code printed on the seal attached to a box in a factory, stable operation can be performed even under conventional reading performance conditions, provided that there are no restrictions on the installation conditions of the optical information reading device 1, thus allowing for optimal installation. Therefore, production cycles can be prioritized without applying image restoration filtering.
[0206] Furthermore, there exists a situation where a process exists where a code clearly printed on metal is read in a factory, and optimal conditions are determined during the setup and adjustment of the reading process. However, during actual operation, the reading becomes unstable due to scratches on the metal surface, changes in the reading position and angle, and variations in ambient brightness. In such cases, applying image restoration filtering to improve image quality increases resistance to these changes in conditions, thereby improving reading performance.
[0207] Furthermore, for certain workpieces, reading using traditional algorithms is impossible. However, in such cases, it becomes possible to read them by applying image restoration filtering. For example, even when the code is significantly scratched or the contrast is unclear, image restoration filtering can be applied to restore the contrast or remove the scratches, thus enabling reading.
[0208] Since the second decoding process, performed when image restoration filtering is applied, decodes the restored image, the time required for this process is typically longer than the time required for the first decoding process, which decodes the normally encoded image. Therefore, the first decoding processing unit 202 can perform the first decoding process in a shorter time than the time taken by the second decoding processing unit 203 to perform the second decoding process. This is used as a premise in this example, and the setting unit 20e can change the decoding timeout, which is a time limit for the decoding process.
[0209] Specifically, the setting unit 20e is configured to set a decoding timeout longer than the time required to perform the second decoding process when the second decoding process is to be performed, and to set a decoding timeout shorter than the time required to perform the second decoding process when the second decoding process is not to be performed. As a result, when the second decoding process is performed during operation, the decoding timeout can be set longer than the time required to perform the normal second decoding process, thus reliably performing the decoding process of the recovered image. On the other hand, when the second decoding process is not performed, typically only the first decoding process, which can be performed in a shorter time than the second decoding process, is performed; therefore, a short decoding timeout corresponding to the first decoding process can be set. Because the decoding timeout is changed in this way, decoding timeouts corresponding to both the first and second decoding processes can be set.
[0210] Despite Figure 17 The first decoding processing unit 202 and the second decoding processing unit 203 are separated, but the first decoding processing unit 202 and the second decoding processing unit 203 can be configured together as follows: Figure 18 The decoding processing unit 210 is shown. The decoding processing unit 210 performs a second decoding process on the recovered image in parallel with the first decoding process performed on the coded image generated by the camera 5. In this way, the parts used to perform the decoding process can be arbitrarily divided in hardware or can be integrated. In the following, an example of integrating the parts used to perform the decoding process into one part will be described, but this aspect is not limited to this and can be equally applied to cases where these parts are divided into multiple parts.
[0211] The decoding processing unit 210 is configured to allocate processing resources intended for the second decoding process to the first decoding process when the setting unit 20e is set not to perform the second decoding process, thereby accelerating the first decoding process compared to when it is set to perform the second decoding process. Processing resources include, for example, the area of memory used and the cores forming a multi-core CPU. Specifically, when the second decoding process is not performed, the number of cores responsible for the first decoding process increases compared to when the second decoding process is performed. Additionally, when the second decoding process is not performed, the area of memory used in the first decoding process is expanded compared to when the second decoding process is performed. Increasing the number of cores and expanding the area of memory can be used in combination, or only one of these methods can be performed. As a result, the performance of the multi-core CPU can be maximized to accelerate the first decoding process.
[0212] Before image restoration, the decoding processing unit 210 extracts candidate regions of the code that are highly likely to contain the code from the code image. That is, if the image input to the neural network is large when performing image restoration using a neural network, the computation time increases. Furthermore, from the viewpoint of accelerating processing, it is desirable to minimize the number of decoding attempts. On the other hand, in this example, a heatmap-based search technique is used instead of line search, so that candidate regions of the code can be reliably extracted even if it takes some time. For example, the decoding processing unit 210 quantizes the feature values of the code, generates a heatmap that assigns the magnitude of the feature values to each pixel value, and extracts candidate regions of the code that are highly likely to contain the code on the heatmap. As a specific example, there is a method for obtaining feature portions (e.g., finder patterns, etc.) of the QR code in relatively hot (with large feature values) regions of the heatmap. In the case of obtaining multiple feature portions, the feature portions can be extracted according to priority and stored in RAM 41 or storage unit 30, etc.
[0213] After the decoding processing unit 210 extracts the code candidate region, it outputs the partial image corresponding to the extracted code candidate region to the image restoration unit 20j. The image restoration unit 20j inputs the partial image corresponding to the code candidate region extracted by the decoding processing unit 210 into the neural network and performs inference processing to generate the restored image obtained by restoring the partial image. As a result, the computation time can be shortened.
[0214] Furthermore, the decoding processing unit 210 can be configured to extract code candidate regions from the code image that have a high probability of containing the size of the code to be read, as set by the aforementioned adjustment processing. After reducing or enlarging the extracted image to a predetermined size, the extracted image is input into a neural network, and the recovered image reconstructed by the neural network is decoded. Since the extracted image is enlarged or reduced to an image of a predetermined size and then input into the neural network, the PPC of the code image input into the neural network can be kept within a predetermined range, and the image reconstruction effect using the neural network can be stably achieved. In addition, it is conceivable that there are various sizes of codes to be read, but the size of the code ultimately input into the neural network can be fixed. Therefore, the balance between high-speed processing and ease of decoding can be maintained at a certain level or higher.
[0215] In the following text, reference will be made to Figure 19 The flowcharts for the first and second decoding processes are described in detail. In step SE1 after the start, the processor 20 causes the camera 5 to generate and acquire a coded image. The coded image acquired in step SE1 is input to the decoding processing unit 210. In step SE2, the first decoding process is performed on the coded image acquired in step SE1 without image restoration. When the first decoding process is successful, the process proceeds to step SE4 to perform a termination process, i.e., the process for outputting the decoding result. If the first decoding process fails in step SE2, the process returns to step SE2 to perform the first decoding process again. When a predetermined decoding timeout has elapsed, the first decoding process for the coded image is stopped.
[0216] In the path from step SE1 to step SE5, the decoding processing unit 210 extracts code regions for recovery from the code image. Specifically, it extracts candidate code regions from the code image, which have a high probability of including the size of the code to be read and the presence of the code, as set by the processing. At this time, when multiple code regions for recovery are extracted, the code regions for recovery are temporarily stored as candidate regions R1, R2, etc. Figure 20 An example of the extracted code region used for recovery is shown below. The process then proceeds to step SE6, and the value of k is set to 1.
[0217] In step SE7, a resizing process is performed on the portion of the image corresponding to the candidate region Rk. This resizing process is used to shrink or enlarge the portion of the image corresponding to the code region extracted in step SE5 for recovery to a predetermined size. Through this resizing process, the size of the image to be input into the neural network described later can be pre-set to a predetermined size. Figure 20The example shown illustrates a case where the size of a portion of the image input to the neural network is 256 pixels × 256 pixels. However, the size of the portion of the image is not limited to this and can be adjusted to any size to take into account factors such as processing speed.
[0218] In step SE8, it is determined whether the black-and-white inversion setting is "both". That is, image restoration involves mathematical mapping, and there are situations where it is difficult to achieve mappings from white to black, from black to white, from black to black, and from white to white using a single neural network. Therefore, black-and-white inversion processing is performed based on attributes to obtain an image that prints black on a white background. Depending on how the neural network learns, it is also possible to print a white image on a black background. An example of black-and-white inversion processing is shown in... Figure 20 As shown in the image. The black-and-white inversion process will be described in detail later.
[0219] If the determination in step SE8 is "yes" and the black-and-white inversion is set to "both", the process proceeds to step SE9, and the resized image is input to the image restoration unit 20j to perform image restoration. Thereafter, the process proceeds to step SE10, and the decoding processing unit 210 performs a second decoding process on the restored image. If the second decoding process is successful in step SE10, the process proceeds to step SE4 to perform termination processing. If the second decoding process fails in step SE10, the process proceeds to step SE11 to perform black-and-white inversion processing on the resized image. Thereafter, in step SE11, the image obtained after the black-and-white inversion processing is input to the image restoration unit 20j to perform image restoration (see [link]). Figure 20 Next, the process proceeds to step SE13, and the decoding processing unit 210 performs a second decoding process on the recovered image. If the second decoding process is successful in step SE13, the process proceeds to step SE4 to perform a termination process. If the second decoding process fails in step SE13, the process proceeds to step SE14. In step SE14, it is determined whether the candidate region extracted in step SE5 still exists, and if no candidate region exists, the process proceeds to step SE4 to perform a termination process.
[0220] On the other hand, if the determination in step SE8 is "No" and the black-and-white inversion setting is not "both", the process proceeds to step SE15 to determine whether the black-and-white inversion setting is "Off". If the determination in step SE15 is "No" and the black-and-white inversion setting is not "Off", the process proceeds to step SE16 to perform black-and-white inversion processing on the resized image. Then, in step SE17, the image obtained after the black-and-white inversion processing is input to the image restoration unit 20j to perform image restoration. Next, the process proceeds to step SE18, and the decoding processing unit 210 performs a second decoding process on the restored image. If the second decoding process is successful in step SE18, the process proceeds to step SE4 to perform termination processing. If the second decoding process fails in step SE18, the process proceeds to step SE14. In step SE14, it is determined whether candidate regions still exist; if no candidate regions exist, the process proceeds to step SE4 to perform termination processing.
[0221] If, in step SE14, it is determined that candidate regions still exist, the process proceeds to step SE19, where the value of k is incremented by one, and then the process proceeds to step SE7. In step SE7, since the value of k has increased by 1 since the previous process, the resizing process is similarly performed on the portion of the image corresponding to the next candidate region Rk. Step SE8 and subsequent steps are applied in the same way as described above. That is, when decoding is successful in either the first or second decoding process, the decoding result is output upon success so that decoding processing is not performed on other candidate regions, and the code image obtained by capturing the next workpiece W is acquired, and the same processing is performed.
[0222] Figure 20 The post-processing unit 201 shown performs a process to combine the portion of the image that has not yet undergone image restoration after resizing with the portion of the image that has undergone image restoration using the image restoration unit 20j after resizing. That is, in image restoration using neural networks, the code before restoration can be visually verified by a human, but situations may arise where image restoration is poor or cells become rounded during restoration. In such cases, superimposing the unrestored portion of the image with the restored portion at an appropriate ratio can sometimes facilitate subsequent decoding. The ratio of the unrestored portion of the image with the restored portion can be a predetermined fixed value or a value other than a fixed value. For example, this ratio can be determined based on the stability of the restoration performed by the image restoration unit 20j or through preliminary evaluation (adjustment) to dynamically optimize the ratio during reading. For example, in preliminary evaluation, the ratio can be adjusted by the user using a scale bar, etc., and can be adjusted during the presentation of the combined image to the user.
[0223] (Library switching function)
[0224] This embodiment is configured such that parameters that form imaging conditions for camera 5 and processing conditions in decoding processing unit 210, etc., are grouped into sets that can be stored in parameter set storage unit 30c. A parameter set can be referred to as a library. Multiple libraries are provided, each storing different parameters. For example, parameter set storage unit 30c stores multiple imaging conditions and code conditions, including first imaging conditions and code conditions, and second imaging conditions and code conditions, set by adjustment execution unit 20d, as different parameter sets.
[0225] The optical information reading device 1 is configured to switch between one parameter set (including a first imaging condition and a code condition) stored in the parameter set storage unit 35c and another parameter set (including a second imaging condition and a code condition), and vice versa. The parameter set switching can also be performed by the processor 20, by the user, or by a switching signal from an external control device such as the PLC 101. In the case of user-controlled parameter set switching, for example, a parameter set switching unit included in the user interface can be operated. When the parameter set switching unit is set to "active," the library's parameter set is used during operation of the optical information reading device 1. When the parameter set switching unit is set to "inactive," the library's parameter set is not used during operation of the optical information reading device 1. In other words, the parameter set switching unit is configured to switch from one parameter set to another.
[0226] When the decoding processing unit 210 in this example fails in the second decoding process of the recovered image while operating with a parameter set, it switches to another parameter set and then performs image recovery and second decoding processes on the code image captured under different conditions. For example, when the decoding processing unit 210 detects that the second decoding process has failed, it performs image recovery and second decoding processes after switching to a parameter set other than the current parameter set. When it detects that the second decoding process has failed even when using the parameter set, it performs image recovery and second decoding processes after switching to another parameter set that has not been used so far. There are cases where switching parameter sets changes not only the imaging conditions of the camera 5 but also the size of the code to be read. In this case, the size of the code candidate region, where the code is highly likely to exist, is also changed. As a result, the reduction or magnification rate of the image input to the neural network also changes, and the result of image recovery also changes.
[0227] (Responding to the black-and-white reversal)
[0228] Next, we will refer to Figures 21A to 22C Handling the black-and-white reversal of the description code. Figures 21A to 21CThis is a view used to describe the use of two neural networks (i.e., a neural network trained using codes printed in black on a white background and a neural network trained using codes printed in white on a black background) to deal with the black-and-white reversal of codes. Figure 21A The scheme illustrates a method where a code image (the leftmost image) is input into two neural networks: one trained using codes printed in black on a white background, and the other trained using codes printed in white on a black background. The respective neural networks output decoding results, and this scheme is referred to as a parallel execution scheme. Since the code image is obtained by photographing codes printed in black on a white background, decoding succeeds in the recovered image obtained using the neural network trained in black on a white background, but fails in the recovered image obtained using the neural network trained in white on a black background.
[0229] exist Figure 21B In this approach, only the code image (the leftmost image) is initially fed into a neural network trained with codes printed in white on a black background to attempt decoding of the recovered image. Only if decoding fails is the code image fed into a neural network trained with codes printed in black on a white background to generate the recovered image. This scheme uses a different network after a failure.
[0230] Figure 21C This describes a scheme where the neural network to which the code image is input is selected based on user settings. The left side shows an image obtained by photographing a code printed in black on a white background. In this case, the network selection module selects a neural network trained using a code printed in black on a white background. The code image (the leftmost image) is then input into the neural network trained using this code. Conversely, the right side shows an image obtained by photographing a code printed in white on a black background. In this case, the network selection module selects a neural network trained using a code printed in white on a black background. The code image (the leftmost image) is then input into the neural network trained using a code printed in white on a black background.
[0231] Figures 22A to 22C It is a view used to describe the case where only one neural network is used to deal with the black-and-white reversal of codes. Figure 22AThis illustrates the generation of a black-and-white inverted image by inverting a code image (the leftmost image). The images, before and after black-and-white inversion, are fed into a neural network trained using codes printed in black on a white background. Since the code image is obtained by photographing codes printed in white on a black background, decoding fails in the recovered image obtained using the neural network trained on black codes on a white background; however, decoding succeeds using the image obtained after black-and-white inversion.
[0232] exist Figure 22B In the example shown at the top, the code image (the leftmost image) is an image of codes printed in white on a black background. Therefore, when the code image is fed into a neural network trained using codes printed in black on a white background, the decoding process fails. Subsequently, when an image that has been reversed in black and white is generated and fed into a neural network trained using codes printed in black on a white background, the decoding process succeeds.
[0233] exist Figure 22B In the example shown at the bottom, the code image (the leftmost image) is an image of codes printed in black on a white background. Therefore, when the code image is fed into a neural network trained using codes printed in white on a black background, the decoding process fails. Subsequently, when an image that has been reversed in black and white is generated and fed into a neural network trained using codes printed in white on a black background, the decoding process succeeds.
[0234] Figure 22C The scheme determines whether to perform black-and-white inversion processing based on user settings. In the case shown above, the code image is obtained by photographing a code printed in black on a white background. In this case, the black-and-white inversion processing module does not perform the inversion, and the code image is directly input into the neural network trained using a code printed in black on a white background. On the other hand, in the case shown below, the code image is obtained by photographing a code printed in white on a black background. In this case, the black-and-white inversion processing module performs the inversion. Then, the image that has undergone black-and-white inversion processing is input into the neural network trained using a code printed in black on a white background.
[0235] (The relationship between contrast and match level)
[0236] Match levels can be used, for example, as scores during adjustment or to manage print quality and read performance during operation. For instance, match levels are defined by positive integer values between 0 and 100 and can be designed to achieve a 100% read rate when tested with a match level of 50 or higher. Additionally, a match level of 0 indicates a read-unreadable condition, and a match level of 1 is the lowest value for a read-readable condition.
[0237] Figure 23 This is a graph showing the relationship between contrast and matching level. The solid line indicates the relationship between the contrast and matching level of the image that has been restored by the image restoration unit 20j (the restored image), and the dashed line indicates the relationship between the contrast and matching level of the image that has not yet been restored by the image restoration unit 20j (the code image). It can be clearly seen from this graph that the matching level of the restored image is generally higher than that of the code image; however, if the image quality deteriorates by a certain level or more, the matching level decreases rapidly.
[0238] (Processing for calculating matching levels)
[0239] A matching level is set for each algorithm. Typically, one algorithm is used for one code type. However, in this embodiment, by installing image restoration functionality, two algorithms (one with image restoration and one without) are executed for one code type. Between these two algorithms, the result of the algorithm whose decoding process succeeded earlier is output. Therefore, when the decoding process alternates between these two algorithms, there is a possibility that the matching level value becomes unstable. Furthermore, since an easily readable image is obtained by performing image restoration, the matching level tends to be higher than existing matching levels in matching level calculation methods similar to conventional methods. On the other hand, in this embodiment, a weighted sum of the matching levels of the two algorithms is used to adjust the matching level value. Referring to the following... Figure 24 The flowchart shown describes a specific example.
[0240] In step SF1 after the start, the coded image is input to the decoding processing unit 210. In step SF2, the decoding processing unit 210 performs a first decoding process without restoring the image. In step SF3, the matching level A (MLA) of the first decoding process is calculated. On the other hand, in step SF4, the coded image is input to the image restoration unit 20j to perform image restoration processing. Thereafter, the process proceeds to step SF5, and the decoding processing unit 210 performs a second decoding process on the restored image. In step SF6, the matching level B (MLB) of the second decoding process is calculated. Note that MLA = 0 is obtained if decoding is unsuccessful in step SF2, and MLB = 0 is obtained if decoding is unsuccessful in step SF5.
[0241] In step SF7, the values of MLA and MLB are adjusted using, for example, an averaging function. Additionally, even if decoding has been successful in step SF2 or step SF5, a weighted processing of the MLA value is performed.
[0242] (Example of a user interface)
[0243] Figure 25 This is a view showing an example of the user interface 300 displayed on the display unit 6. The processor 20 generates the user interface 300 and displays it on the display unit 6. The user interface 300 is provided with a first image display area 301, a second image display area 302, and a third display area 303. The first image display area 301 displays the image currently captured by the camera 5 (live view image), the second image display area 302 displays the decoded image, and the third display area 303 displays the recovered image recovered by the image recovery unit 20j. As a result, the decoded image and the recovered image can be presented to the user in a comparable form. In addition, the user can know what type of image is being decoded or what image is obtained after decoding.
[0244] (Example of an imaging element with an AI chip)
[0245] Figures 26A to 26C This is a diagram illustrating an example of using an imaging element 5a with an AI chip. The imaging element 5a of the camera 5 can be configured as shown in the various figures. Figure 26A The diagram shows an imaging element 5a equipped with an AI chip for performing image restoration and encapsulated therein, where the coded image is restored and then output to the processor 20. Figure 26B The diagram shows an imaging element 5a equipped with an AI chip that performs code region extraction and image restoration for recovery, and is encapsulated therein. The code region for recovery is extracted from the code image, and then the portion of the image corresponding to that region is restored and output to the processor 20. Figure 26C The diagram shows an imaging element 5a equipped with an AI chip that performs the extraction of code regions for recovery and is encapsulated therein, where code regions for recovery are extracted from the code image, and then a portion of the image corresponding to that region is output to the processor 20.
[0246] exist Figure 26A In the example shown, the code image acquired by imaging element 5a can be output to processor 20, the AI chip can perform image restoration, and the restored image can be output to processor 20. The same applies to... Figure 26B and Figure 26C The example shown.
[0247] (Polarization filter accessory 3)
[0248] In this embodiment, as Figure 2 As shown, a first illumination section 4a and a second illumination section 4b, each formed by multiple light-emitting diodes, are configured as light sources to generate illumination light for at least one illumination code. The polarizing filter accessory 3 includes a first polarizer 3a and a second polarizer 3b. The first polarizer 3a allows light with a first polarization component to pass through the light generated by the light-emitting diodes forming the second illumination section 4b, and the second polarizer 3b allows light with a second polarization component that is substantially orthogonal to the first polarization component to pass through. The first polarizer 3a and the second polarizer 3b can be respectively disposed in… Figure 2 Within the shadow line range. For example, the first polarizer 3a is arranged to cover the second illumination part 4b, and the second polarizer 3b is arranged to cover the optical system 5b of the camera 5 from the light incident side.
[0249] Therefore, camera 5 receives light that has passed through first polarizer 3a and reflected from the surface (code) of workpiece W via second polarizer 3b. As a result, the specular reflection component of workpiece W is removed, and camera 5 generates a code image with lower contrast compared to the case where neither first polarizer 3a nor second polarizer 3b is inserted. Processor 20 acquires such a low-contrast code image. Processor 20 inputs this low-contrast code image into a neural network to convert the code image into a recovered image with higher contrast than before input, and performs decoding processing on the recovered image.
[0250] In other words, by using polarizers 3a and 3b to remove the specular reflection component of the workpiece W, a code image with reduced specular reflection component influence can be obtained. For example, polarizers 3a and 3b are suitable for images with many specular reflection components, such as those obtained by photographing a metal workpiece, but there is a situation where the code image becomes darker and the contrast decreases due to the reduction in light intensity. In this case, reading performance is improved by using inference processing via a neural network to convert the code image into a recovered image with high contrast.
[0251] The above embodiments are merely examples of all aspects and should not be interpreted in a limiting manner. Furthermore, all modifications and alterations falling within the equivalent scope of the claims are within the scope of this aspect.
[0252] As described above, the optical information reading device according to this aspect can be used, for example, to read codes attached to a workpiece.
Claims
1. An optical information reading device for reading a code attached to a workpiece, the optical information reading device comprising: A camera, used to capture an image of the code and generate a code image; The storage unit is used to store the structure and parameters of a neural network pre-generated by performing machine learning on multiple defective images having portions unsuitable for code reading and multiple ideal images corresponding to the multiple defective images respectively; The processor is configured to execute a first decoding process for decoding a coded image generated by the camera, to execute an inference process for inputting the coded image generated by the camera into a neural network configured using structures and parameters stored in the storage unit to recover the coded image based on the structures and parameters stored in the storage unit, and to execute a second decoding process on the recovered coded image, wherein the inference process and the first decoding process are executed in parallel; and A housing for accommodating the camera, the storage unit, and the processor. The processor is configured as follows: Extract code candidate regions with a high probability of the code's existence from the code image. Inference processing for generating the recovered image is performed by inputting a portion of the image corresponding to the code candidate region into the neural network. The grid positions indicating the locations of individual cells of the code are determined based on the recovered image, and Decoding is performed on the recovered image based on the grid positions. At least a portion of the time period during which the processor performs the second decoding process has a pause period during which the inference process is not performed.
2. The optical information reading device according to claim 1, further comprising: The filtering processing unit is used to perform image processing filtering on the code image generated by the camera before it is input into the neural network.
3. The optical information reading device according to claim 2, wherein, The filtering unit performs noise removal filtering to remove noise included in the image generated by the camera.
4. The optical information reading device according to claim 1, wherein, The processor includes an integrated circuit dedicated to attempting to recover the code image.
5. The optical information reading device according to claim 1, wherein, The processor is configured to perform decoding processing on the code image before attempting to recover it, attempt to recover the code image if decoding fails, and perform decoding processing on the recovered code image.
6. The optical information reading device according to claim 1, further comprising: The setting section is used to configure whether to perform the second decoding process. The processor is capable of executing the first decoding process in a shorter time than the time required for the second decoding process, and The setting unit is configured to set a decoding timeout longer than the time required to perform the second decoding process when the second decoding process is to be performed, and to set a decoding timeout shorter than the time required to perform the second decoding process when the second decoding process is not to be performed.
7. The optical information reading device according to claim 1, further comprising: The setting section is used to configure whether to perform the second decoding process. When the setting unit is set not to perform the second decoding process, the processor allocates processing resources intended for the second decoding process to the first decoding process, and accelerates the first decoding process compared to when the setting unit is set to perform the second decoding process.
8. The optical information reading device according to claim 1, further comprising: The image restoration unit is used to input the coded image generated by the camera into a neural network configured using the structure and parameters stored in the storage unit, and to perform inference processing for generating a restored image obtained by restoring the coded image; The decoding processing unit is used to perform decoding processing on the recovered image generated by the image restoration unit; as well as The adjustment execution unit is used to repeat image capturing and decoding processing while changing the imaging conditions of the camera and the decoding conditions of the decoding process. It determines the optimal imaging and decoding conditions based on a matching level of the ease of reading the indicator code calculated under each imaging and decoding condition, and performs adjustment processing to set the size of the code to be read. The decoding processing unit extracts a code candidate region that has a high probability of containing the code to be read, which is set by the adjustment processing. It then shrinks or enlarges the extracted image to make the image have a predetermined size. The shrunken or enlarged image is input into the neural network, and the recovered image is decoded by the neural network.
9. The optical information reading device according to claim 1, further comprising: Multiple light sources are used to generate illumination light for at least illuminating the code; A first polarizer is used to allow light of a first polarization component in the light generated by the light source to pass through; and The second polarizer is used to allow light of the second polarization component, which is orthogonal to the first polarization component, to pass through. The camera receives light that has passed through the first polarizer and reflected from the code via the second polarizer, and removes the specular reflection component of the workpiece to generate a code image with lower contrast compared to the case where neither the first nor the second polarizer is inserted. The processor inputs a low-contrast code image into the neural network to convert the code image into a recovered image with higher contrast than before the input, and performs decoding processing on the recovered image.
10. The optical information reading device according to claim 1, further comprising: The image restoration unit is used to input the coded image generated by the camera into a neural network configured using the structure and parameters stored in the storage unit, and to perform inference processing for generating a restored image obtained by restoring the coded image; The decoding processing unit is used to perform decoding processing on the recovered image generated by the image restoration unit; as well as The adjustment actuator is capable of repeatedly capturing images and performing decoding processing while changing the imaging conditions of the camera, and is capable of setting multiple imaging conditions and code conditions for the code to be read based on a matching level of the ease of reading the indicator code calculated under each of the imaging conditions. The storage unit stores multiple imaging conditions and code conditions set by the adjustment execution unit, including a first imaging condition and code condition, and a second imaging condition and code condition. The decoding processing unit inputs the code image generated under the first imaging condition and code condition from among the multiple reading conditions stored in the storage unit into the neural network to generate the recovered image, and if the decoding processing of the recovered image fails, it inputs the code image generated under the second imaging condition and code condition into the neural network to generate the recovered image, and performs decoding of the recovered image.
11. An optical information reading device for reading a code attached to a workpiece, the optical information reading device comprising: A camera, used to capture an image of the code and generate a code image; The storage unit is used to store the structure and parameters of a neural network pre-generated by performing machine learning on multiple defective images having portions unsuitable for code reading and multiple ideal images corresponding to the multiple defective images respectively; A setting unit is configured to set an image restoration filter for attempting to recover the code image based on the structure and parameters stored in the storage unit; The processor is configured to execute a first decoding process for decoding a coded image generated by the camera, to execute an inference process for applying an image restoration filter set by the setting unit to the coded image generated by the camera to attempt to recover the coded image, and to execute a second decoding process on the recovered coded image obtained after the attempt to recover the image, wherein the inference process and the first decoding process are executed in parallel; and A housing for accommodating the camera, the storage unit, and the processor. The processor is configured as follows: Extract code candidate regions with a high probability of the code's existence from the code image. Inference processing for generating the recovered image is performed by inputting a portion of the image corresponding to the code candidate region into the neural network. The grid positions indicating the locations of individual cells of the code are determined based on the recovered image, and Decoding is performed on the recovered image based on the grid positions. At least a portion of the time period during which the processor performs the second decoding process has a pause period during which the inference process is not performed.
12. The optical information reading device according to claim 11, wherein, The settings unit is configured to receive user settings for the image restoration filter.
13. The optical information reading device according to claim 11, further comprising: The adjustment execution unit is used to set various conditions when setting the optical information reading device, so that the various conditions become suitable for the decoding process. The adjustment execution unit includes the setting unit, which is configured to set the image restoration filter.
14. The optical information reading device according to claim 13, wherein, The adjustment execution unit sets parameters related to the image restoration filtering.
15. The optical information reading device according to claim 13, wherein, The adjustment actuator is configured with code conditions in the code image generated by the camera.
16. The optical information reading device according to claim 15, wherein, The size of the code image to be input into the neural network is set based on the code conditions set by the adjustment execution unit.
17. The optical information reading device according to claim 11, wherein, The storage unit is configured to store a set of parameters including information related to the settings of the image restoration filter. The storage unit stores a first set of parameters for setting the image restoration filter and a second set of parameters for not setting the image restoration filter, and The processor performs decoding processing by applying a parameter set selected from the first parameter set and the second parameter set.
Citation Information
Patent Citations
Identification code reading apparatus and machine learning device
CN109614830A
Two-dimensional code deformation recovery method, device and equipment
CN110490023A
