Learning data generation method, learning model, information processing device, and information processing method

Abstract

In this learning data generation method, a first code image is generated, which is a graphic code. In the learning data generation method, the first code image is subjected to first image processing to generate a second code image. In the learning data generation method, the second code image is superimposed on a background image to generate a first image.

Description

Learning data generation method, learning model, information processing device, and information processing method CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Japanese Patent Application No. 2022-119999, filed on July 27, 2022, the entire disclosure of which is incorporated herein by reference.

[0002] The present invention relates to a training data generation method, a training model, an information processing device, and an information processing method.

[0003] It is known that a barcode encoding information identifying a product is attached to the packaging of a product, etc. The encoded information can be decoded by scanning the barcode with a dedicated barcode reader such as a barcode scanner or touch scanner (see Patent Document 1).

[0004] JP 2015-153224 A

[0005] A training data generation method according to a first aspect generates a first code image which is a code of a graphic shape, generates a second code image by performing a first image processing on the first code image, and generates a first image by superimposing the second code image on a background image.

[0006] Furthermore, the learning model according to the second aspect generates a first code image which is a symbol of a figure, generates a second code image by performing a first image processing on the first code image, generates a first image by superimposing the second code image on a background image, and generates a second image by performing a second image processing on the first image, the second image being a change in at least one of the color and contrast of the entire first image, and causes a computer to function to output a region of a partial image of a symbol of a figure in the input image after the computer has been trained using the second image generated by a learning data generation method.

[0007] Furthermore, the learning model according to the third aspect generates a first code image which is a symbol of a figure shape, generates a second code image by performing a first image processing on the first code image, and causes a computer to function so as to output an area of ​​a partial image of a symbol of a figure shape in an input image, after the computer has been trained using the first image generated by a learning data generation method which generates a first image by superimposing the second code image on a background image.

[0008] Further, an information processing device according to a fourth aspect includes: an acquisition unit that acquires a captured image; and a control unit that extracts a partial image of a graphical code by inputting the captured image into a detection model and decodes the code based on the extracted partial image, wherein the detection model is a learning model that causes a computer to function to output, for an input image, an area of ​​the partial image of the graphical code in the input image, having been trained using a second image generated by a learning data generation method that generates a first code image that is a graphical code, generates a second code image by performing first image processing on the first code image, generates a first image by superimposing the second code image on a background image, and generates a second image by performing second image processing on the first image, wherein the second image processing is a change in at least one of color and contrast of the entire first image.

[0009] Further, an information processing method according to a fifth aspect includes acquiring a captured image, inputting the captured image into a detection model to extract a partial image of a graphical code, and decoding the code based on the extracted partial image, wherein the detection model generates a first code image that is a graphical code, generates a second code image by performing first image processing on the first code image, generates a first image by superimposing the second code image on a background image, and generates a second image by performing second image processing on the first image, the second image processing being a change in at least one of color and contrast of the entire first image, and is a learning model that causes a computer to function to output, for an input image, an area of ​​the partial image of the graphical code in the image.

[0010] A sixth aspect of the training data generation method includes generating a first code image used to identify attributes of an item, generating a second code image by performing a first image processing on the first code image, and generating a training image by superimposing the second code image on a background image including an image of the item and a package containing the item, wherein the first image processing is a process for reproducing an appearance in which at least one of the item and the package containing the item is visible through the first code image.

[0011] 7 is a configuration diagram showing a schematic configuration of an information processing system including an information processing device according to an embodiment. FIG. 8 is a block diagram showing a schematic configuration of the terminal device of FIG. 1. FIG. 9 is a block diagram showing a schematic configuration of a first information processing device of FIG. 2. FIG. 10 is a first diagram for explaining processing for decoding a code in the first information processing device of FIG. 3. FIG. 11 is a second diagram for explaining processing for decoding a code in the first information processing device of FIG. 3. FIG. 12 is a block diagram showing a schematic configuration of the second information processing device of FIG. 2. FIG. 13 is a block diagram showing a schematic configuration of a third information processing device that executes a training data generation method according to an embodiment. FIG. 14 is an image showing an example of a third code image. FIG. 15 is an image showing another example of the third code image. FIG. 16 is a flowchart for explaining decoding processing executed by the control unit of FIG. 3. FIG. 17 is a flowchart for explaining training data generation processing executed by the control unit of FIG.

[0012] Conventionally, decoding graphical codes such as barcodes requires reading the codes with a dedicated scanner or capturing an image of the codes with a camera. However, both reading and capturing the codes require maintaining the codes in an appropriate position and orientation relative to the scanner or camera. Such alignment and orientation adjustment requires the operator's skill. Furthermore, decoding graphical codes is required to be performed quickly, and there has been a demand for code decoding that can be performed quickly and easily. In view of these circumstances, an object of the present disclosure is to perform decoding of graphical codes quickly, easily, and with high accuracy.

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following drawings, the same components are denoted by the same reference numerals.

[0014] 1 , an information processing system 10 including a first information processing device (information processing device) according to an embodiment of the present disclosure is configured to include at least one terminal device 11, a network 12, and a second information processing device 13. In this embodiment, the information processing system 10 includes a plurality of terminal devices 11. The terminal device 11 and the second information processing device 13 may communicate with each other via the network 12.

[0015] The information processing system 10 is applicable to any system that identifies a detection target based on an image of the detection target included in an image. The information processing system 10 is applicable, for example, to a checkout system that identifies a product that is a detection target based on an image. The information processing system 10 will be described below using an example in which the information processing system 10 is applied to a checkout system.

[0016] The information processing system 10 applied to the checkout system is used for checking out merchandise. A graphic code is attached to the surface or packaging of a merchandise. The graphic code is a graphic code obtained by encoding specific information about the merchandise based on an arbitrary encoding algorithm. The graphic code is, for example, a one-dimensional code such as a barcode, or a two-dimensional code such as a QR code (registered trademark). The specific information about the merchandise may be information that identifies the merchandise, such as the merchandise name or an identification number assigned to each merchandise.

[0017] The terminal device 11 may capture an image of the product. The terminal device 11 may detect and decode a graphical code in the image generated by capturing the image. The terminal device 11 may recognize specific information about the product through the decoding. The second information processing device 13 may calculate a billing amount based on the specific information about the product. The terminal device 11 may present the billing amount to the purchaser and request payment of the purchase amount.

[0018] As shown in FIG. 2, the terminal device 11 may include an imaging unit 14, an output device 15, a mounting base 16, a support column 17, and a first information processing device 18.

[0019] The imaging unit 14 is fixed, for example, so as to be able to image at least a portion of the range of the mounting table 16. The imaging unit 14 is fixed, for example, to a support column 17 extending from the side of the mounting table 16. The imaging unit 14 is fixed, for example, so as to be able to image the entire upper surface us of the mounting table 16 and so that its optical axis is perpendicular to the upper surface us. The imaging unit 14 may capture moving images. In other words, the imaging unit 14 may continuously generate captured images at a predetermined frame rate. The captured images may be analog signals or digital data.

[0020] The imaging unit 14 may be configured to include a visible light or infrared camera. The camera includes an imaging optical system and an imaging element. The imaging optical system includes optical components such as one or more lenses and an aperture. The lens may be of any type regardless of focal length, and may be, for example, a general lens, a wide-angle lens including a fisheye lens, or a zoom lens with a variable focal length. The imaging optical system forms an image of a subject on the light receiving surface of the imaging element. The imaging element may be, for example, a charge coupled device (CCD) image sensor, a complementary metal-oxide semiconductor (CMOS) image sensor, an FIR (far infrared ray) camera, or the like. The imaging element captures the subject image formed on the light receiving surface to generate a captured image.

[0021] The output device 15 may be any conventionally known display that displays an image. The display may function as a touch screen, as described below. The output device 15 may be a speaker that notifies information. For example, the output device 15 may output information identifying a product obtained by decoding a graphical code in the first information processing device 18. The output device 15 may issue various notifications when a malfunction occurs in the information processing system 10, etc. If the output device 15 fails to decode the graphical code, it may output an instruction to change the orientation of the product. The output device 15 may output the success or failure of the decoding by the first information processing device 18 and an instruction to change the position of the product.

[0022] 3 , the first information processing device 18 includes a communication unit 19 (acquisition unit) and a control unit 20. The first information processing device 18 may further include a storage unit 21 and an input unit 22. In this embodiment, the first information processing device 18 is configured as a device separate from the imaging unit 14 and the output device 15, but may be configured integrally with at least one of the imaging unit 14, the mounting base 16, the support column 17, and the output device 15, for example.

[0023] The communication unit 19 includes, for example, a communication module that communicates with the imaging unit 14 via a communication line that may be wired or wireless. The communication unit 19 acquires an image from the imaging unit 14. The communication unit 19 may include a communication module that communicates with the output device 15 via the communication line. The communication unit 19 may transmit an image to be displayed as an image signal to the output device 15. The communication unit 19 may receive a position signal corresponding to a position where a contact is detected on the display surface from the output device 15, which is a display. The communication unit 19 may include a communication module that communicates with the second information processing device 13 via the network 12. The communication unit 19 may receive parameters for constructing a detection model, as described below, from the second information processing device 13. The parameters may be analog signals or digital data. The communication unit 19 may transmit decoded product identification information, as described below, to the second information processing device 13. The product identification information may be analog signals or digital data. The communication unit 19 may receive amount information corresponding to a billing amount from the second information processing device 13. The amount information may be an analog signal or digital data.

[0024] The input unit 22 is capable of detecting an operation input from a user. The input unit 22 includes at least one input interface capable of detecting an input from a user. The input interface is, for example, a physical key, a capacitance key, a pointing device, a touch screen integrated with a display, a microphone, etc. In this embodiment, the input / output interface is a touch screen using the output device 15.

[0025] The storage unit 21 includes any of semiconductor memory, magnetic memory, and optical memory. The semiconductor memory is, for example, RAM (Random Access Memory) or ROM (Read Only Memory). The RAM is, for example, SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory). The ROM is, for example, EEPROM (Electrically Erasable Programmable Read Only Memory). The storage unit 21 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 21 stores data used in the operation of the first information processing device 18 and data obtained by the operation of the first information processing device 18. For example, the storage unit 21 stores system programs, application programs, embedded software, etc. For example, the storage unit 21 stores parameters for constructing a detection model obtained from the second information processing device 13.

[0026] The control unit 20 is configured to include at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for specific processing. The dedicated circuit may be, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 20 executes processing related to the operation of the first information processing device 18 while controlling each unit of the first information processing device 18.

[0027] 4, the control unit 20 may store the captured image ci acquired via the communication unit 19 in the storage unit 21. The control unit 20 may generate a low-resolution image lri by reducing the resolution of the entire captured image ci already stored in the storage unit 21. The control unit 20 may reduce the resolution of the captured image ci using known image processing such as an LPF (Low Pass Filter). The low-resolution image lri may be an analog signal or digital data.

[0028] The control unit 20 inputs the captured image ci into the detection model to detect a region pia of a partial image of a graphical symbol in the captured image ci. Alternatively, the control unit 20 inputs the low-resolution image lri instead of the captured image ci into the detection model to detect a region pia of a partial image of a graphical symbol in the low-resolution image lri. The detection model may detect a graphical symbol within the entire image and estimate the region occupied by the symbol. The region occupied by the symbol may be an analog signal or digital data. The detection model may be a learning model, as described below. As shown in FIG. 5 , the control unit 20 extracts, for example, a partial image of the region pia from the captured image ci stored in the storage unit 21. The control unit 20 may extract multiple regions pia. The partial images may be analog signals or digital data. The control unit 20 decodes the graphical symbol based on the partial image in the extracted region pia.

[0029] When the reliability of detection of the region pia of the partial image of the code using the detection model is equal to or lower than a reliability threshold, the control unit 20 may cause the output device 15 to output a request to orient the graphical code toward the imaging unit 14. Alternatively, when the control unit 20 cannot decode the graphical code based on the partial image in the extracted region pia, the control unit 20 may cause the output device 15 to output a request to orient the graphical code toward the imaging unit 14. Note that, for example, in a configuration in which the control unit 20 has a large processing capacity and has ample capacity to detect the partial region pia of the graphical code, the control unit 20 may decode the code in the captured image ci based on the input of the detection model without using the low-resolution image lri, as described above.

[0030] As shown in FIG. 6 , the second information processing device 13 may include a communication unit 23 , a storage unit 24 , and a control unit 25 .

[0031] The communication unit 23 may include at least one communication module connectable to the network 12. The communication module is, for example, a communication module compatible with communication standards such as a wired LAN (Local Area Network), a wireless LAN, Wi-Fi, etc. The communication unit 23 may be connected to the network 12 via a wired LAN or the like by the communication module.

[0032] The communication unit 23 may include a communication module capable of communicating with various external devices via a communication line. The communication module is a communication module that complies with the standards of the communication line. The communication line may be configured to include at least one of a wired and a wireless communication line.

[0033] The memory unit 24 includes any of semiconductor memory, magnetic memory, and optical memory. The semiconductor memory is, for example, RAM or ROM. The RAM is, for example, SRAM or DRAM. The ROM is, for example, EEPROM. The memory unit 24 may function as a main memory device, an auxiliary memory device, or a cache memory. The memory unit 24 stores data used in the operation of the second information processing device 13. For example, the memory unit 24 stores system programs, application programs, embedded software, etc. Furthermore, for example, the memory unit 24 stores the sales price of each product registered in the production system.

[0034] The control unit 25 is configured to include at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for a specific process. The dedicated circuit may be, for example, an FPGA, an ASIC, or the like. The control unit 25 executes processes related to the operation of the second information processing device 13 while controlling each unit of the second information processing device 13.

[0035] When the control unit 25 acquires product identification information from the terminal device 11 through decryption, the control unit 25 may read the sales price of the product corresponding to the identification information from the storage unit 24. The control unit 25 may calculate the billing amount by adding up the sales prices of the products. The control unit 25 may transmit amount information corresponding to the billing amount to the terminal device 11 to which the product identification information has been assigned.

[0036] The detection model used in the first information processing device 18 is a learning model that is trained using, as training data, a combination of an image including, as a partial image, an image of a product having a graphic symbol attached to its surface and information indicating the position of the symbol. A method for generating training data from training data will be described below.

[0037] The learning data may be generated by, for example, a third information processing device 26 as shown in Fig. 7. The third information processing device 26 may be a general-purpose information processing device such as a personal computer (PC) or a server device, or a dedicated information processing device. The third information processing device 26 may include an input / output interface 27, an output unit 28, an input unit 29, a storage unit 30, and a control unit 31.

[0038] The input / output interface 27 inputs and outputs data to and from, for example, a camera or other information processing device, directly or indirectly via a network. For example, the input / output interface 27 may acquire character information for generating a first code image, which is a graphical code. The character information may be an analog signal or digital data. The character information may be any information, may be meaningful information, or may be meaningless information consisting simply of a list of characters, etc.

[0039] The input / output interface 27 may also acquire a third code image that has been generated by capturing an image of an actual graphic code. The third code image may be an analog signal or digital data. The third code image preferably includes images of a code captured not only from the front but also from various directions. The third code image preferably includes an image of a code drawn on a flexible package and captured in a curved and deformed state, as shown in FIG. 8, for example. The third code image preferably includes an image of a code that has been blurred by capturing an image shifted from the focused position, for example. The third code image preferably includes an image of a graphic code through which an underlying image can be seen, as shown in FIG. 9, for example.

[0040] The input / output interface 27 may also acquire a background image. The background image may be an analog signal or digital data. The background image is a wide-area image including an object on which a graphic symbol is drawn, such as a product or product packaging. The wide-area image may be, for example, an image of an object placed on a stand or the like together with the seat surface of the stand.

[0041] The output unit 28 may include one or more interfaces that output information to notify the user, such as, but not limited to, a display that outputs information visually or a speaker that outputs information audibly.

[0042] The input unit 29 may include one or more interfaces for detecting user input, such as physical keys, capacitive keys, and a touch screen that is integrated with the display of the output unit 28.

[0043] The storage unit 30 includes any of semiconductor memory, magnetic memory, and optical memory. The semiconductor memory is, for example, RAM or ROM. The RAM is, for example, SRAM or DRAM. The ROM is, for example, EEPROM. The storage unit 30 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 30 stores data used in the operation of the third information processing device 26. For example, the storage unit 30 stores system programs, application programs, embedded software, and the like. For example, the storage unit 30 may store character information, a third code image, and a background image acquired via the input / output interface 27.

[0044] The control unit 31 is configured to include at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for a specific process. The dedicated circuit may be, for example, an FPGA, an ASIC, or the like. The control unit 31 executes processes related to the operation of the third information processing device 26 while controlling each unit of the third information processing device 26.

[0045] The control unit 31 generates a first code image which is a graphic code. The first code image may be an analog signal or digital data. The control unit 31 may generate the first code image by encoding character information acquired by the input / output interface 27 and the input unit 29. Alternatively, the control unit 31 may generate character information by randomly listing a predetermined number of characters, and encode the character information to generate the first code image. The control unit 31 may perform encoding using any encoding algorithm.

[0046] The control unit 31 may generate learning data using a third code image separately from the first code image. When the control unit 31 uses the third code image to generate learning data, the control unit 31 may determine the number of first code images to be generated based on the number of third code images to be acquired. The number of first code images and the number of third code images are the numbers of codes that can be decoded into independent specific information. The number of first code images to be generated may be the same as the number of third code images.

[0047] The control unit 31 generates a second code image by performing first image processing on the first code image. The second code image may be an analog signal or digital data. The first image processing may be processing that reproduces deformations such as distortion and curvature of the graphical code that occur due to deformations such as distortion and curvature of the object or package bearing the graphical code. The first image processing may be processing that reproduces deformations of the graphical code that occur when the object or package bearing the graphical code is placed on the placing table 16. The first image processing may be processing that reproduces various orientations of the graphical code relative to the imaging device 14 that occur when the object or package bearing the graphical code is placed on the placing table 16. The first image processing may be processing that reproduces blur due to deviation from the focus position of the imaging device 14 caused by the size of the object or package bearing the graphical code, and various sizes of the graphical code caused by the distance from the imaging device 14. The first image processing may be processing that reproduces transparency caused by the material of the object or package bearing the graphical code. The first image processing may be a process of reproducing the current state of an item or package with a graphic symbol attached thereto according to the surrounding environment when the item or package is placed on the placing table 16, such as some blown-out highlights due to reflection from a light source, on the graphic symbol.

[0048] Therefore, the first image processing may include at least one of rotation, enlargement or reduction, distortion, blurring, transparency, and local discoloration. The first image processing may be a combination of at least two of rotation, enlargement or reduction, distortion, blurring, transparency, and local discoloration. The distortion may be a process of reproducing an appearance in which the graphical code is changed in accordance with the three-dimensional shape, such as curvature, of the article on which the graphical code is drawn. The distortion may be a process of reproducing an appearance in which the graphical code is changed in accordance with the three-dimensional shape, such as distortion or curvature, of flexible packaging on which the graphical code is drawn, as shown in FIG. 8 . The transparency may be a process of reproducing an appearance in which the graphical code is visible through the graphical code, as shown in FIG. 9 . Furthermore, the transparency may be a process of reproducing an appearance in which the graphical code appears to be mixed with pictures or letters drawn on the article or packaging, by changing the transparency of at least a part of the first code image in consideration of the material to which the graphical code is applied. The local discoloration is, for example, a process for increasing brightness, such as the glossiness of a surface on which a graphic code is drawn in an actual captured image ci. The first image processing may be the same for all first code images, or may be different for all first code images.

[0049] The control unit 31 may generate second code images by performing first image processing on the third code images. The first image processing may be the same for all third code images, or may be different for each third code image. When the control unit 31 has already generated first code images in excess of the number of third code images, the control unit 31 may adjust the total number of second code images based on the first code images to be the same as the total number of second code images based on the third code images by increasing the number of second code images generated for each third code image to be greater than the number of second code images generated for each first code image.

[0050] The control unit 31 generates a first image by superimposing the second code image on the background image. The first image may be an analog signal or digital data. The control unit 31 may superimpose the second code image, particularly, on an object in the background image. The object may include, for example, an item. The object may include, for example, a package (an item and packaging containing the item). The packaging may be a packaging film or a packaging container that allows the contained item to be visually recognized. The packaging may also be wrapping paper, packaging film, or a packaging container that prevents the contained item from being visually recognized. When the control unit 31 generates a second code image that has been subjected to transparency as the first image processing and the background image includes a package, the control unit 31 may generate a first image in a manner that allows one or both of the item and the packaging containing the item to be visually recognized in the background of the second code image. The control unit 31 may generate a second code image through transparency as the first image processing. When the background image includes a package wrapped so that the contained items can be seen, the control unit 31 may generate a first image in a manner that allows the items inside the package and other objects placed underneath the package to be seen. When the control unit 31 performs processing including transparency as the first image processing to generate a second code image and superimpose it on the object, the control unit 31 may generate a first image in a manner that the second code image and the object are mixed together. The control unit 31 may superimpose multiple background images on each second code image to generate multiple first images for each second code image. The control unit 31 may generate a first image based on a third code image. The first image based on the third code image may include an image generated by superimposing the second code image, generated by performing the first image processing on the third code image, on the background image. Alternatively, the first image based on the third code image may include an image generated by superimposing the third code image on the background image without performing the first image processing on the third code image. When superimposing the second code image on the background image, the control unit 31 may recognize position information indicating the area where the second code image is to be superimposed.

[0051] The control unit 31 may generate a second image by performing second image processing on the first image. The second image may be an analog signal or digital data. The second image processing may be processing that reproduces changes caused by the model of the imaging device 16 and settings at the time of shooting. It may be processing that reproduces changes caused by settings of the imaging device 16 at the time of shooting. The second image processing may be processing that reproduces changes in appearance due to the illuminance value when the imaging device 16 captures an image of a graphical code. Therefore, the second image processing is processing that changes at least one of the color and contrast of the entire first image. The color change refers to, for example, a change in hue, saturation, or brightness. The second image processing may be the same for all second code images, or may be different.

[0052] The control unit 31 may associate the generated second image with position information indicating the area in the second image where the second code image is superimposed as teacher data, and store the associated data in the storage unit 30. In generating the teacher data, for the purpose of easily increasing the number of teacher data while improving the accuracy of detection, the control unit 31 may associate the generated first image with position information indicating the area in the first image where the second code image is superimposed as teacher data, and store the associated data in the storage unit 30.

[0053] Next, the decoding process executed by the control unit 20 of the first information processing device 18 in this embodiment will be described with reference to the flowchart in Fig. 10. The decoding process starts every time the communication unit 19 of the first information processing device 18 acquires one frame of the captured image ci.

[0054] In step S100, the control unit 20 stores the acquired captured image ci in the storage unit 21. After storage, the process proceeds to step S101.

[0055] In step S101, the control unit 20 generates a low-resolution image lri by reducing the resolution of the entire captured image. After generation, the process proceeds to step S102.

[0056] In step S102, the control unit 20 detects a region pia of a partial image of a graphical symbol by inputting the low-resolution image lri generated in step S101 into the detection model. After detection, the process proceeds to step S103.

[0057] In step S103, the control unit 20 extracts a partial image in the area at the same position as the area pia detected in step S102 from the captured image ci stored in the storage unit 21 in step S100. After extraction, the process proceeds to step S104.

[0058] In step S104, the control unit 20 determines whether the reliability of the detection in step S102 is equal to or less than the reliability threshold. If it is equal to or less than the reliability threshold, the process proceeds to step S107. If it is not equal to or less than the reliability threshold, the process proceeds to step S105.

[0059] In step S105, the control unit 20 decodes the graphic code based on the partial image extracted in step S103. After decoding, the process proceeds to step S106.

[0060] In step S106, the control unit 20 determines whether the decoding in step S105 has failed. If the decoding has failed, the process proceeds to step S107. If the decoding has succeeded, the decoding process ends.

[0061] In step S107, the control unit 20 controls the output device 15 to output a request to direct the graphic code toward the imaging unit 14. After the output, the decoding process ends.

[0062] Next, the learning data generation process executed by the control unit 31 of the third information processing device 26 in this embodiment will be described with reference to the flowchart in Fig. 11. The learning data generation process starts when the input unit 29 detects an operation input for generating learning data.

[0063] In step S200, the control unit 31 acquires third code images from a camera or other information processing device via the input / output interface 27, or from the storage unit 30. Furthermore, the control unit 31 counts the number of acquired third code images. After counting, the process proceeds to step S201.

[0064] In step S201, the control unit 31 determines the number of first code images to be created based on the number of third code images counted in step S200. After the determination, the process proceeds to step S202.

[0065] In step S202, the control unit 31 generates first code images by encoding character information. The control unit 31 generates the first code images using character information acquired by another information processing device via the input / output interface 27, character information acquired by the input unit 29, character information stored in the storage unit 30, or character information generated by the control unit 31. The control unit 31 generates the number of first code images determined in step S201. After generation, the process proceeds to step S203.

[0066] In step S203, the control unit 31 generates a second code image by performing the first image processing on the third code image acquired in step S200 and the first code image generated in step S202. After generation, the process proceeds to step S204.

[0067] In step S204, the control unit 31 generates a first image by superimposing the second code image generated in step S203 on a background image. The control unit 31 may acquire the background image from another information processing device or a camera via the input / output interface 27, or from the storage unit 30. After generation, the process proceeds to step S205.

[0068] In step S205, the control unit 31 recognizes the position information of the area in the first image generated in step S204 where the second code image is superimposed. After the recognition, the process proceeds to step S206.

[0069] In step S206, the control unit 31 generates a second image by performing second image processing on the first image generated in step S204. After generation, the process proceeds to step S207.

[0070] In step S207, the control unit 31 stores the position information recognized in step S205 and the second image generated in step S206 in the storage unit 30 in association with each other.

[0071] The third information processing device 26 of this embodiment configured as described above generates a first code image, performs first image processing on the first code image to generate a second code image, and generates a first image by superimposing the second code image on a background image. The decoding accuracy of a graphical code using the first information processing device 18 described above increases with the detection accuracy of the detection model. The detection accuracy of the detection model can be improved by learning using a large amount of supervised data. However, creating the supervised data requires the operator to identify the area of ​​the graphical code after capturing an image of the object, which places a significant burden on the operator. On the other hand, the third information processing device 26 configured as described above does not require capturing an image of the graphical code and identifying its position, and can generate a large amount of supervised data for learning with low load. Therefore, the third information processing device 26 can improve the detection accuracy of the detection model, thereby contributing to quick and easy decoding of the graphical code by the first information processing device 18.

[0072] Furthermore, in the third information processing device 26, the first image processing is at least one of rotation, enlargement or reduction, distortion, blurring, transparency, and local discoloration. When the imaging unit 14 captures an image of a graphical code to be decoded, the partial image of the code contains various sizes, various orientations, distortion due to curvature of the product, distortion due to distortion and curvature of the packaging, blurring due to deviation from the focus position, transparency depending on the material to which the graphical code is attached, and partial highlighting due to reflection from a light source. In response to such phenomena, the third information processing device 26 having the above configuration can generate a second code image that reflects phenomena that may be contained in the code of the actual partial image. Therefore, the third information processing device 26 can generate training data that further improves the detection accuracy of the detection model.

[0073] Furthermore, in the third information processing device 26, the first image processing changes the transmittance of the first code image to generate a second code image in which the first code image is mixed with a background image. When the imaging unit 14 captures an image of a graphical code to be decoded, depending on the material to which the graphical code is attached, images of the background item and the packaging containing the item are included in the partial image of the code due to transparency. For such an event, the third information processing device 26 having the above configuration can generate a second code image that reflects the event that may be included in the code of the actual partial image. Therefore, the third information processing device 26 can generate learning data that further improves the detection accuracy of the detection model.

[0074] Furthermore, in the third information processing device 26, the background image includes an image of an article, and the first image processing generates a second code image in which a graphical code is changed according to the three-dimensional shape of the article. When the imaging unit 14 captures the graphical code to be decoded, a partial image is generated in which the code also suffers from distortion due to the distortion or curvature of the article. For such an event, the third information processing device 26 having the above configuration can generate a second code image that reflects the event that may be included in the code of the actual partial image. Therefore, the third information processing device 26 can generate learning data that further improves the detection accuracy of the detection model.

[0075] Furthermore, in the third information processing device 26, the background image includes an image of the packaging, and the first image processing generates a second code image in which the graphical code is changed according to the three-dimensional shape of the packaging. When the imaging unit 14 captures the graphical code to be decoded, a partial image is generated in which the code also suffers from distortion due to distortion or curvature of the packaging. For such an event, the third information processing device 26 having the above configuration can generate a second code image that reflects the event that may be included in the code of the actual partial image. Therefore, the third information processing device 26 can generate learning data that further improves the detection accuracy of the detection model.

[0076] Furthermore, the third information processing device 26 generates a second image by performing a second image processing on the first image, the second image processing being a change in at least one of the color and contrast of the entire first image. When the imaging unit 14 captures a graphic code to be decoded, the appearance of the code may change depending on the illumination light illuminating the graphic. In response to such an event, the third information processing device 26 having the above configuration can generate a second image that reflects an event that may be included in an actual partial image. Therefore, the third information processing device 26 can generate training data that further improves the detection accuracy of the detection model.

[0077] Moreover, the third information processing device 26 acquires the third code image and generates the first image based on the third code image. With this configuration, the third information processing device 26 can generate learning data that improves the detection accuracy of the area of ​​the code in an image obtained by capturing an image of a commodity or the like that actually includes the graphic code.

[0078] Moreover, the third information processing device 26 determines the number of first code images to be generated based on the number of third code images to be acquired. With this configuration, the third information processing device 26 prevents the first code images from being generated without limit relative to the number of third code images to be generated by actual imaging. Therefore, the third information processing device 26 can generate learning data that further improves the detection accuracy of the area of ​​a graphic code in an image obtained by imaging a commodity or the like that actually includes the code.

[0079] Moreover, the third information processing device 26 generates a second code image by performing the first image processing on the third code image. With this configuration, the third information processing device 26 generates an image by deforming an actually captured image, and therefore can provide a variety of learning data. Therefore, the third information processing device 26 can generate learning data that further improves the detection accuracy of the detection model.

[0080] In this embodiment, the first information processing device 18 includes a communication unit 19 that acquires the captured image ci, and a control unit 20 that extracts a partial image of a graphical code by inputting the captured image ci into a detection model and decodes the code based on the extracted partial image, the detection model being the learning model described above. Since the first information processing device 18 performs decoding based on the captured image ci, it does not require a dedicated scanner and does not require alignment or orientation adjustment, making it possible to easily decode the code.

[0081] In addition, the first information processing device 18 detects a partial image area pia in the low-resolution image lri, which is obtained by reducing the resolution of the captured image ci, by inputting the low-resolution image lri into the detection model by the control unit, and decodes the code based on the detected partial image. Since the first information processing device 18 uses the low-resolution image lri to detect the partial area of ​​the graphical code, detection can be performed quickly.

[0082] In one embodiment, (1) a learning data generation method includes generating a first code image, which is a code of a graphic shape; generating a second code image by performing a first image processing on the first code image; and generating a first image by superimposing the second code image on a background image.

[0083] (2) In the training data generation method of (1), the first image processing is at least one of rotation, enlargement or reduction, distortion, blurring, transparency, and local discoloration.

[0084] (3) In the learning data generation method of (1) or (2) above, the background image includes an image of an item and a package containing the item, and the first image processing generates the second code image in a manner that allows one or both of the item and the package, which form the background, to be visible through the symbol on the graphic.

[0085] (4) In the training data generation method of (3) above, the first image processing generates the second code image in which the first code image is mixed with the background image by changing the transparency of the first code image.

[0086] (5) In the learning data generation method of (1) to (4) above, the background image includes an image of an object, and the first image processing generates the second code image by changing the graphical code according to the three-dimensional shape of the object.

[0087] (6) In the learning data generation method of (1) to (5) above, the background image includes an image of packaging, and the first image processing generates the second code image by changing the graphical symbol according to the three-dimensional shape of the packaging.

[0088] (7) In the training data generation method of (1) to (6) above, a second image is generated by performing a second image processing on the first image, and the second image processing is a change in at least one of the color and contrast of the entire first image.

[0089] (8) In the learning data generating method of (1) to (7) above, a third code image, which is a code of a graphic shape, is acquired, and the first image is generated based on the third code image.

[0090] (9) In the learning data generating method of (8) above, the number of the first code images to be generated is determined based on the number of the third code images to be acquired.

[0091] (10) In the learning data generating method of (8) or (9), the second code image is generated by performing the first image processing on the third code image.

[0092] (11) In the learning data generating method of (8) or (9) above, the first image is generated by superimposing the third code image on a background image.

[0093] (12) A learning model to which the learning data generation method of (7) above is applied causes a computer to function so as to output a region of a partial image of a graphical symbol in an input image, after learning using a second image generated by the learning data generation method.

[0094] (13) A learning model to which the learning data generation methods of (1) to (11) above are applied causes a computer to function so as to output a region of a partial image of a graphical symbol in an input image, after learning using a first image generated by the learning data generation method.

[0095] In one embodiment, (14) an information processing device includes: an acquisition unit that acquires a captured image; and a control unit that inputs the captured image into a detection model to extract an area of ​​a partial image of a graphical symbol and decodes the symbol based on the extracted partial image, wherein the detection model is the learning model described in (12) or (13).

[0096] (15) In the information processing device of (14) above, the control unit detects the area of ​​the partial image in the low-resolution image by inputting a low-resolution image obtained by reducing the resolution of the captured image into the detection model, and decodes the code based on the detected partial image.

[0097] In one embodiment, (16) an information processing method includes: acquiring a captured image; inputting the captured image into a detection model to extract a partial image of a graphical symbol; and decoding the symbol based on the extracted partial image; and the detection model is a learning model described in (12) or (13).

[0098] In one embodiment, (17) a learning data generation method includes generating a first code image used to identify attributes of an item, generating a second code image by performing a first image processing on the first code image, and generating a learning image by superimposing the second code image on a background image including the item and a package containing the item, wherein the first image processing is a process of reproducing an appearance in which at least one of the item and the package containing the item can be seen through the first code image.

[0099] The above has described embodiments of the first information processing device 18 and the third information processing device 26, but embodiments of the present disclosure can also be embodied as a method or program for implementing the device, as well as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card, etc.).

[0100] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module incorporated into an operating system. Furthermore, the program may or may not be configured so that all processing is performed solely by the CPU on the control board. The program may also be configured so that part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as needed.

[0101] The drawings illustrating the embodiments of the present disclosure are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0102] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art could make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions included in each component can be rearranged so as not to cause logical inconsistencies, and multiple components can be combined or divided into one.

[0103] For example, in the above-described embodiment, the control unit 31 in the third information processing device 26 generates the second image based on the first code image and the third code image. However, the control unit 31 may generate the second image based on only either the first code image or the third code image.

[0104] For example, in the above-described embodiment, the control unit 31 in the third information processing device 26 generates the second image by performing the second image processing on the first image, which is obtained by superimposing a background image on the second code image after the first image processing. However, the control unit 31 may generate the second image by further performing the second image processing on the second code image without superimposing a background image on the second code image.

[0105] For example, in the above-described embodiment, a first code image is generated for the purpose of detecting a code on a graphic. However, the control unit 31 may generate a first code image of a code used to identify the attributes of an item, such as a price tag or a discount sticker, generate a second code image by performing a first image processing on the first code image, and generate a learning image by superimposing the second code image on a background image. In this case, a combination of an image including, as a partial image, a product having a code used to identify the attributes of the item attached to its surface and information indicating the position of the code may be used as training data.

[0106] All of the features described in this disclosure and / or all steps of all of the disclosed methods or processes may be combined in any combination except combinations in which these features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by an alternative feature serving the same, equivalent, or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is only one example of a generic series of identical or equivalent features.

[0107] Furthermore, embodiments of the present disclosure are not limited to the specific configurations of any of the above-described embodiments, but rather extend to any novel feature or combination thereof described herein, or any novel method or process step or combination thereof described herein.

[0108] In the present disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. Configurations distinguished by descriptions such as "first" and "second" in the present disclosure can exchange numbers in the configuration. For example, a first information processing device can exchange identifiers "first" and "second" with a second information processing device. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the identifier exchange. Identifiers may be deleted. Configurations from which identifiers have been deleted are distinguished by symbols. The descriptions of identifiers such as "first" and "second" in the present disclosure should not be used solely to interpret the order of the configurations or to justify the existence of identifiers with smaller numbers.

[0109] REFERENCE SIGNS LIST 10 Information processing system 11 Terminal device 12 Network 13 Second information processing device 14 Imaging unit 15 Output device 16 Placement base 17 Support column 18 First information processing device (information processing device) 19 Communication unit (acquisition unit) 20 Control unit 21 Memory unit 22 Input unit 23 Communication unit 24 Memory unit 25 Control unit 26 Third information processing device 27 Input / output interface 28 Output unit 29 Input unit 30 Memory unit 31 Control unit ci Captured image lri Low-resolution image pia Area of ​​partial image of symbol of graphic shape us Top surface

Claims

1. generating a first code image which is a code of a graphic shape; generating a second code image by performing a first image processing on the first code image; A first image is generated by superimposing the second code image on a background image. Data generation method.

2. 2. The data generating method according to claim 1, The first image processing is at least one of rotation, enlargement or reduction, distortion, blurring, transparency, and local discoloration. Data generation method.

3. 3. The data generating method according to claim 1, the background image includes an image of an item and a package containing the item; The first image processing generates the second code image in a manner that allows one or both of the item and the packaging, which are background, to be visible through the symbol on the figure. Data generation method.

4. 4. The data generating method according to claim 3, the first image processing generates the second code image in which the first code image is mixed with the background image by changing a transparency of the first code image; Data generation method.

5. 3. The data generating method according to claim 1, the background image includes an image of an item; The first image processing generates the second code image by changing the graphic code according to the three-dimensional shape of the article. Data generation method.

6. 3. The data generating method according to claim 1, the background image includes an image of a package; The first image processing generates the second code image by changing the graphic code in accordance with the three-dimensional shape of the package. Data generation method.

7. 3. The data generating method according to claim 1, generating a second image by performing a second image processing on the first image; The second image processing is a change in at least one of color and contrast across the first image. Data generation method.

8. 3. The data generating method according to claim 1, A third code image, which is a code of a graphic shape, is obtained; generating the first image based on the third code image Data generation method.

9. A learning model for causing a computer to function so as to output a region of a partial image of a graphical symbol in an input image, the learning model being trained using a second image generated by the data generation method described in claim 7.

10. A learning model for causing a computer to function so as to output a region of a partial image of a graphical symbol in an input image, the learning model being trained using a first image generated by the data generation method described in claim 1 or 2.

11. An acquisition unit that acquires a captured image; a control unit that inputs the captured image into a detection model to extract a partial image of a graphical code and decodes the code based on the extracted partial image, The detection model is a learning model according to claim 9. Information processing device.

12. 12. The information processing device according to claim 11, The control unit detects an area of ​​the partial image in the low-resolution image by inputting a low-resolution image obtained by reducing the resolution of the captured image to the detection model, and decodes the code based on the detected partial image. Information processing device.

13. Acquire the captured image, The captured image is input to a detection model to extract a partial image of a graphic symbol; Decoding the code based on the extracted partial image; The detection model is a learning model according to claim 9. Information processing methods.

14. generating a first code image used to identify an attribute of the article; generating a second code image by performing a first image processing on the first code image; A method for generating a learning image by superimposing the second code image on a background image including an image of the item and a package containing the item, comprising: The first image processing is a processing for reproducing an aspect in which at least one of the item and the packaging containing the item can be visually recognized through the first code image. Data generation method.