Processing system, processing method, and program
The system enhances product recognition accuracy by using images with incorrect results for retraining only when a threshold is met, addressing the computational overload issue in existing systems.
Patent Information
- Application Number
- JP2024017648
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-02-08
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2040-05-20
Smart Images

Figure 0007578205000001 
Figure 0007578205000002 
Figure 0007578205000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a processing system, a processing method, and a program. [Background technology]
[0002] Patent Document 1 discloses a technology for recognizing products based on images of the products. Non-Patent Document 1 discloses a multi-object recognition technology that combines product recognition by feature point matching and product recognition to which deep learning is applied. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2016-062545 A [Non-patent literature]
[0004] [Non-Patent Document 1] "Multi-object recognition technology that can recognize any retail product", [online], [Retrieved April 27, 2020], Internet<URL: https: / / jpn.nec.com / techrep / journal / g19 / n01 / 190118.html> Summary of the Invention [Problem to be solved by the invention]
[0005] It is expected to improve the accuracy of image-based product recognition. Therefore, the inventors have studied a technology for accumulating images that are input to an estimation model as images to be analyzed (images containing products to be recognized) during actual operation in a store or the like as training data, and updating the estimation model by re-learning using the training data.
[0006] The state of the product in the image to be analyzed (the orientation, shading, shape, size, etc. of the product) changes depending on the shooting environment, etc. In the case of the above technology, the image that actually became the image to be analyzed during actual operation in a store, etc. can be used as training data, so that the above re-learning generates an estimation model suitable for actual operation in the store, etc., and improves the accuracy of product recognition during actual operation in the store, etc. In addition, since the images input to the estimation model during actual operation in the store, etc. can be accumulated as training data, the effort of collecting training data is saved.
[0007] However, during actual operation in a store or the like, the amount of images input to the estimation model as images to be analyzed (images containing the product to be recognized) is enormous even in just one day. Furthermore, if actual operation in a store or the like continues for a long period of time, the amount of accumulated images will further increase. If all of these are used as training data, the processing burden on the computer will increase. Naturally, the more frequently re-learning is performed, the greater the processing burden on the computer will be.
[0008] An object of the present invention is to improve the accuracy of image-based product recognition while reducing the processing burden on a computer that generates an estimation model. [Means for solving the problem]
[0009] According to the present invention, An image acquisition means for acquiring a recognition processing image including a recognition target product; A recognition means for recognizing a product in the recognition processing image based on an estimation model generated by machine learning; a registration means for registering the result of the recognition in recognized product information; an output means for outputting the result of the recognition; a correction receiving means for receiving an input for correcting the result of the recognition; a correction means for changing the recognition result registered in the recognized product information to a corrected recognition result, and storing correction information linking the corrected recognition result with the recognition processing image in a storage means; a learning means for updating the estimation model by re-learning using the recognition processing images stored as the correction information when the number of the recognition processing images stored as the correction information exceeds a predetermined value; A processing system is provided having:
[0010] Further, according to the present invention, The computer Acquire a recognition processing image including the product to be recognized; Recognizing the product in the recognition processing image based on an estimated model generated by machine learning; The result of the recognition is registered in the recognized product information; Outputting the results of the recognition; Accepting input to correct the recognition result; changing the recognition result registered in the recognized product information to the corrected recognition result, and storing correction information linking the corrected recognition result with the recognition processing image in a storage means; There is provided a processing method in which, when the number of the recognition processing images stored as the correction information exceeds a predetermined value, re-learning is performed using the recognition processing images stored as the correction information to update the estimation model.
[0011] Further, according to the present invention, Computer, An image acquisition means for acquiring a recognition processing image including a recognition target product; A recognition means for recognizing a product in the recognition processing image based on an estimation model generated by machine learning; a registration means for registering the result of the recognition in recognized product information; an output means for outputting the result of the recognition; a correction receiving means for receiving an input for correcting the result of the recognition; a correction means for changing the recognition result registered in the recognized product information to a corrected recognition result, and storing correction information linking the corrected recognition result with the recognition processing image in a storage means; a learning means for updating the estimation model by re-learning using the recognition processing images stored as the correction information when the number of the recognition processing images stored as the correction information exceeds a predetermined value; A program is provided to function as a Effect of the Invention
[0012] According to the present invention, it is possible to improve the accuracy of image-based product recognition while reducing the processing load on a computer that generates an estimation model. [Brief description of the drawings]
[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of a hardware configuration of a processing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a functional block diagram of a processing system according to an embodiment of the present invention; [Diagram 3] 1 is an example of an implementation of a transaction device included in the processing system of this embodiment. [Figure 4] FIG. 2 is a functional block diagram of a processing system according to an embodiment of the present invention; [Diagram 5] FIG. 4 is a diagram illustrating an example of information processed by the processing system of the present embodiment. [Figure 6] FIG. 13 is a diagram showing an example of a screen output by the processing system of the present embodiment. [Figure 7] FIG. 4 is a diagram illustrating an example of information processed by the processing system of the present embodiment. [Figure 8] 4 is a flowchart showing an example of a process flow of the processing system of the present embodiment. [Figure 9] FIG. 2 is a functional block diagram of a processing system according to an embodiment of the present invention; [Figure 10] FIG. 2 is a diagram for explaining an example of camera installation according to the present embodiment. [Figure 11] FIG. 2 is a diagram for explaining an example of camera installation according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] <First embodiment> "overview" The processing system of this embodiment accumulates, as training data, only "images for which the recognition results are incorrect" among images input to the estimation model as images to be analyzed (images containing products to be recognized) during actual operation in a store, etc. Then, when the amount of training data accumulated under such conditions exceeds a predetermined value, the processing system re-learns based on the training data accumulated up to that point and updates the estimation model. This will be described in detail below.
[0015] "Hardware Configuration" Next, an example of the hardware configuration of the processing system will be described.
[0016] Each functional part of the processing system is realized by any combination of hardware and software, centered on the CPU (Central Processing Unit) of any computer, memory, programs loaded into the memory, a storage unit such as a hard disk that stores the programs (in addition to programs that are stored before the device is shipped, it can also store programs downloaded from storage media such as CDs (Compact Discs) or servers on the Internet), and a network connection interface.Those skilled in the art will understand that there are many variations in the methods and devices for realizing this.
[0017] FIG. 1 is a block diagram illustrating a hardware configuration of a processing system. As shown in FIG. 1, the processing system has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The processing system does not have to have the peripheral circuit 4A. The processing system may be composed of a plurality of physically and / or logically separated devices, or may be composed of a single device that is physically and / or logically integrated. When the processing system is composed of a plurality of physically and / or logically separated devices, each of the plurality of devices can have the above hardware configuration.
[0018] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to transmit and receive data to each other. The processor 1A is an arithmetic processing system such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The input / output interface 3A includes an interface for acquiring information from an input device, an external device, an external server, an external sensor, a camera, etc., and an interface for outputting information to an output device, an external device, an external server, etc. Examples of the input device include a keyboard, a mouse, a microphone, a physical button, a touch panel, etc. Examples of the output device include a display, a speaker, a printer, a mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.
[0019] "Function Configuration" The processing system 10 of this embodiment is an accounting system as shown in Fig. 2. The accounting system has an accounting device operated by an operator and a server that communicates with multiple accounting devices. That is, the processing system 10 has an accounting device and a server.
[0020] The accounting device is a device used at the time of payment in a store, and executes a registration process to register products to be paid for. The accounting device may further execute a settlement process to settle the payment amount. The accounting device may be a device that is assumed to be operated by a store clerk, or may be a device that is assumed to be operated by a customer.
[0021] In the registration process, the checkout device acquires product identification information for the product that is the subject of the checkout. The checkout device then acquires product information (product name, unit price, etc.) linked to the acquired product identification information from a store server or the like, and stores the information in the device's own storage device as transaction information.
[0022] Acquisition of product identification information is achieved by product recognition based on an image. That is, when the accounting device acquires an image including a product, it recognizes the product included in the image and acquires product identification information of the recognized product. Taking a picture of an image including a product is achieved by an operation by an operator (a store clerk or a customer).
[0023] The accounting device may also accept input of product identification information using well-known technology via input devices such as a code reader, touch panel, physical buttons, microphone, keyboard, or mouse.
[0024] In the settlement process, the accounting device executes a process to settle the transaction amount. The accounting device can use any payment method, such as credit card payment, cash payment, points payment, code payment, etc. If the accounting device does not execute the settlement process, it can send the registered transaction information (information on the products subject to the transaction, the transaction amount, etc.) to a settlement device that executes the settlement process.
[0025] An example of the implementation of the accounting device is shown in Fig. 3. Note that the implementation shown is merely an example and is not limited to this. The accounting device has a stand 101, a product placement area 102, a support 103, a camera 104, a computer 105, a touch panel display 106, and a code reader 107.
[0026] An operator places one or more items to be processed on the product placement area 102. Multiple items can be placed on the product placement area 102 at one time. A camera 104 is attached to a support 103 in a position and orientation that allows the camera 104 to photograph the product placement area 102. The camera 104 photographs one or more items placed on the product placement area 102 all at once.
[0027] The camera 104 and the computer 105 can communicate with each other by any means. An image generated by the camera 104 is input to the computer 105 by real-time processing. The code reader 107 and the computer 105 can communicate with each other by any means. Information acquired by the code reader 107 is input to the computer 105 by real-time processing. The touch panel display 106 and the computer 105 can communicate with each other by any means. Information acquired by the touch panel display 106 is input to the computer 105 by real-time processing. Although not shown, the accounting device may be equipped with other input devices such as a microphone, physical buttons, a keyboard, a mouse, etc. These input devices and the computer 105 can communicate with each other by any means. Information acquired by these input devices is input to the computer 105 by real-time processing.
[0028] The computer 105 executes various processes based on the acquired information, and can display the results of the processes on the touch panel display 106.
[0029] Note that while the accounting device in this implementation example is configured to photograph multiple items at once, as a variant, the accounting device may be configured to photograph each item one by one as the operator positions each item in front of the camera.
[0030] 4 shows an example of a functional block diagram of a processing system 10 including an accounting device and a server. As shown in the figure, the processing system 10 has an image acquisition unit 11, a recognition unit 12, a registration unit 13, an output unit 14, a memory unit 15, a correction acceptance unit 16, a correction unit 17, and a learning unit 18. For example, the accounting device has the image acquisition unit 11, the recognition unit 12, the registration unit 13, the output unit 14, the memory unit 15, the correction acceptance unit 16, and the correction unit 17. And the server has the learning unit 18.
[0031] The image acquisition unit 11 acquires a recognition processing image that is an image including a recognition target product. The image acquisition unit 11 acquires an image generated by the camera 104 in FIG.
[0032] The recognition unit 12 recognizes products in the recognition processing image based on an estimation model generated by machine learning, and outputs product identification information (product code, etc.) of the recognized products.
[0033] The estimation model is, for example, a classifier to which deep learning is applied. More specifically, the estimation model may be a model to which the multi-object recognition technology disclosed in Non-Patent Document 1 is applied. The recognition unit 12 recognizes products in the recognition processing image by inputting the recognition processing image to the estimation model. The recognition processing image input to the estimation model may be an image including the entire recognition processing image, or may be an image obtained by cutting out a partial area in which an object is detected in the recognition processing image. For example, in the case of the configuration shown in FIG. 3, a plurality of products may be included in one recognition processing image. In this case, for example, the recognition unit 12 performs object recognition processing on the recognition processing image, and then generates a plurality of images by cutting out each of the object areas detected in the recognition processing image. Then, the recognition unit 12 inputs each of the cut-out images to the estimation model to recognize each of the plurality of products in the recognition processing image.
[0034] The estimation model outputs, for example, the reliability that the input image includes products of each of a plurality of classes. The recognition unit 12 identifies one class based on the reliability of each of the plurality of classes, and outputs product identification information of the identified class as a recognition result. For example, the recognition unit 12 may identify the "class with the highest reliability," or may identify the "class with the highest reliability and whose reliability is equal to or greater than a reference value," or may identify one class by combining the reliability and other parameters, or may identify one class by other methods.
[0035] 2, the registration unit 13 registers the recognition result (product identification information) output by the recognition unit 12 in the recognized product information. In this embodiment, the recognized product information is transaction information that indicates products that have been registered as transaction objects.
[0036] An example of the recognized product information is shown in Fig. 5. For example, the storage unit 15 stores the recognized product information.
[0037] In the illustrated example, a serial number for identifying registered products from each other, a product code and product name which are product identification information of the registered product, the unit price of the registered product, and the image file name of the recognition processing image including the registered product are linked to each other.
[0038] When the registration unit 13 acquires the product identification information output by the recognition unit 12, it acquires product information (product name, unit price, etc.) linked to the acquired product identification information from a store server or the like, and registers it in the recognized product information as shown in Fig. 5. The store server stores a product master in advance.
[0039] Furthermore, the registration unit 13 links each recognition result registered in the recognized product information with the recognition processing image on which each recognition result is based, and stores the recognition processing image on which each recognition result is based in the storage unit 15. The recognition processing image on which each recognition result is based is an image input to the estimation model, and is an image including the entire recognition processing image, or an image obtained by cutting out a partial area in which an object is detected in the recognition processing image.
[0040] 2, the output unit 14 outputs the recognition result to the operator. The output unit 14 can display a list of a plurality of recognition results registered in the recognized product information (see FIG. 5).
[0041] Fig. 6 shows a schematic example of a screen output by the output unit 14. For example, the screen shown in Fig. 6 is displayed on the touch panel display 106 of Fig. 3. Note that the output unit 14 may also project a screen such as that shown in Fig. 6 at an arbitrary position via a projection device. The projection position may be any position that can be viewed by an operator operating the processing system 10.
[0042] Returning to Fig. 2, the correction accepting unit 16 accepts an input to correct the recognition result. The correction accepting unit 16 accepts an input to specify one of the multiple recognition results displayed in a list as shown in Fig. 6, and an input to correct the specified recognition result. The input to correct the specified recognition result is the input of correct product identification information (product code, product name, etc.). Any input device such as a touch panel, microphone, mouse, keyboard, physical button, code reader, etc. can be used as a means for realizing these inputs.
[0043] For example, the operator views a screen such as that shown in Fig. 6 to check whether there are any erroneous recognition results. If an erroneous recognition result is found, the operator inputs information specifying the erroneous recognition result and correct product identification information. By configuring the system so that correct product identification information can be input via a code reader, input errors can be avoided.
[0044] 2, the correction unit 17 changes the recognition result registered in the recognized product information to the corrected recognition result. That is, the correction unit 17 changes the recognition result designated by the input accepted by the correction accepting unit 16, among the recognition results registered in the recognized product information, to the correct product identification information indicated by the input accepted by the correction accepting unit 16.
[0045] Furthermore, the correction unit 17 stores correction information linking the corrected recognition result (the correct product identification information indicated by the input accepted by the correction accepting unit 16) with the recognition processing image on which the erroneous recognition result before correction was based in the storage unit 15. The recognition processing image on which the erroneous recognition result before correction was based is an image input to the estimation model, and is an image including the entire recognition processing image, or an image obtained by cutting out a partial area in which an object was detected in the recognition processing image.
[0046] An example of the correction information is shown in Figure 7. The correction information shown in the figure is linked to each product code that is the result of the recognition after correction, and stores the recognition processing image that was the basis of the erroneous recognition result before the correction.
[0047] Returning to Fig. 2, when the number of recognition processing images stored as correction information exceeds a predetermined value (design item), the learning unit 18 re-learns using the recognition processing images stored as correction information to update the estimation model. The number of recognition processing images stored as correction information is preferably counted for each product (for each product code in the example of Fig. 7), but may be counted for each product group of the same type of product, or may be counted for all products together.
[0048] Next, an example of the flow of processing performed by the processing system 10 will be described using the flowchart in Figure 8. As described above, the processing system 10 has a payment device and a server, and the flowchart in Figure 8 shows an example of the flow of processing performed by the payment device.
[0049] First, the image acquisition unit 11 acquires a recognition processing image including a product to be recognized (S10). For example, an operator places a product to be checked out on the product placement area 102 in Fig. 3. Then, the image acquisition unit 11 acquires a recognition processing image including the product placed on the product placement area 102, which is generated by the camera 104.
[0050] Next, the recognition unit 12 recognizes the product in the recognition processing image acquired in S10 based on the estimation model generated by machine learning (S11). Then, the recognition unit 12 outputs, as a recognition result, product identification information of the product estimated to be included in the recognition processing image.
[0051] Next, the registration unit 13 registers the recognition results (product identification information) output by the recognition unit 12 in the recognized product information (see FIG. 5) (S12). In this embodiment, the recognized product information is transaction information indicating the products registered as transaction items. The registration unit 13 also acquires product information (product name, unit price, etc.) linked to the acquired product identification information from a store server or the like, and registers it in the recognized product information. The registration unit 13 also stores the recognition processing image on which each recognition result was based in the memory unit 15, linking it to each of the recognition results registered in the recognized product information.
[0052] Next, the output unit 14 outputs the recognition result to the operator (S13). For example, the output unit 14 displays a screen as shown in Fig. 6, which displays a list of multiple recognition results registered in the recognized product information (see Fig. 5), on the touch panel display 106 of Fig. 3.
[0053] After outputting the recognition result to the operator, the correction accepting unit 16 becomes able to accept an input to correct the recognition result. The correction accepting unit 16 accepts an input to specify one of the multiple recognition results displayed in a list as shown in Fig. 6, and an input to correct the specified recognition result. For example, the correction accepting unit 16 accepts an input to specify one recognition result to be corrected via the touch panel display 106 of Fig. 3 displaying a screen as shown in Fig. 6. Also, for example, the correction accepting unit 16 accepts an input of correct product identification information via the code reader 107 of Fig. 3.
[0054] Then, when the correction receiving unit 16 receives an input to correct the recognition result (Yes in S14), the correction unit 17 changes the recognition result registered in the recognized product information to the corrected recognition result (S15). That is, the correction unit 17 changes the recognition result to be corrected, which is specified by the input received by the correction receiving unit 16, among the recognition results registered in the recognized product information, to the correct product identification information indicated by the input received by the correction receiving unit 16.
[0055] Furthermore, the correction unit 17 stores in the storage unit 15 the correction information linking the result of the recognition after the correction (the correct product identification information indicated by the input accepted by the correction accepting unit 16) with the recognition processing image on which the erroneous recognition result before the correction was based (S16). Note that the processing order of S15 and S16 is not limited to that shown in the figure.
[0056] Although not shown in the flowchart of Figure 8, the processing system 10 can receive an instruction input to execute a settlement process at any timing thereafter. For example, the processing system 10 receives an instruction input to execute a settlement process by receiving an operation to touch the "Checkout (Settlement)" button on a screen such as that shown in Figure 6. In response to the instruction input, the processing system 10 executes the settlement process or sends the registered transaction information (information on the product subject to the transaction, the transaction amount, etc.) to a settlement device that executes the settlement process.
[0057] Although not shown, at a predetermined timing, the server of the processing system 10 judges whether the number of recognition processing images stored as correction information exceeds a predetermined value. If it is judged that the number exceeds the predetermined value, the processing system 10 re-learns using the recognition processing images stored as correction information to update the estimation model. On the other hand, if it is judged that the number does not exceed the predetermined value, the processing system 10 does not execute re-learning at that timing. The predetermined timing may be a timing at which a predetermined time has passed, a timing at which the operator inputs an instruction to execute the judgment, or any other timing.
[0058] "Action and effect" The processing system 10 of this embodiment accumulates, as training data, only "images for which the recognition results are incorrect" among images input to the estimation model as images to be analyzed (images including a product to be recognized) during actual operation in a store, etc. Then, when the amount of training data accumulated under such conditions exceeds a predetermined value, the processing system 10 re-learns based on the training data accumulated up to that point and updates the estimation model.
[0059] According to such a processing system 10, during actual operation in a store or the like, it is possible to use as training data not all of the images input into the estimation model as images to be analyzed (images including the product to be recognized), but rather an appropriately selected set of images from those images, thereby reducing the processing load on the computer required to update the estimation model.
[0060] Furthermore, according to the processing system 10 of this embodiment, since "images with incorrect recognition results" can be used as training data for re-learning, such errors are less likely to occur due to re-learning. In other words, the effect of re-learning can be improved.
[0061] In addition, since the processing system 10 performs re-learning when the amount of accumulated teacher data exceeds a predetermined value, it is possible to avoid re-learning at unnecessary times when the amount of accumulated teacher data is small and the effect of re-learning is not sufficient. As a result, the processing load on the computer required to update the estimation model is reduced.
[0062] <Second embodiment> The processing system 10 of this embodiment has a function of specifying the correction target in the displayed list of recognition results by image analysis. This can eliminate the operator's work of specifying the correction target. This will be described in detail below.
[0063] The correction receiving unit 16 receives input of correct product identification information via a code reader as input for correcting the recognition result. For example, an operator who checks a screen displaying a list of recognition results as shown in Fig. 6 and finds an incorrect recognition result identifies the product among the items of the transaction that is not correctly registered (the product with the incorrect recognition result). The operator then inputs the correct product identification information of the product identified via a code reader (such as code reader 107 in Fig. 3).
[0064] In this embodiment, the camera is installed at a position and orientation that captures the process of having the code reader read the code attached to the product. The camera in question may be the same camera as the camera (such as camera 104 in FIG. 3) that captures the product to be checked out, or it may be a different camera.
[0065] The image acquisition unit 11 acquires a corrected image, which is an image generated by a camera that captures the process of having a code reader read a code attached to a product.
[0066] The correction unit 17 identifies the object of correction among the recognition results registered in the recognized product information based on the correction image. For example, the recognition unit 12 recognizes the product included in the correction image based on the same estimation model as the estimation model used to recognize the product in the recognition processing image. Then, the correction unit 17 can identify the recognition results registered in the recognized product information that match the recognition results of the correction image as the object of correction.
[0067] As another example, the correction unit 17 may specify the correction target based on the similarity between the appearance feature of the product in the recognition processing image and the appearance feature of the product in the correction image. In this example, the recognition result of the recognition processing image having the highest similarity to the appearance feature of the product in the correction image among the recognition results registered in the recognized product information is specified as the correction target.
[0068] The other configurations of the processing system 10 of this embodiment are similar to those of the first embodiment.
[0069] According to the processing system 10 of this embodiment, the same action and effect as the processing system 10 of the first embodiment is realized. Moreover, according to the processing system 10 of this embodiment, the correction target can be automatically identified by image analysis, so that the operator's work of specifying the correction target can be omitted. In this way, a user-friendly configuration is realized.
[0070] As a modification of this embodiment, the processing system 10 may use the corrected image as training data. The corrected image includes a product whose recognition result is incorrect. By using such a corrected image as training data, it is possible to efficiently increase the training data of the product whose recognition result is incorrect.
[0071] <Third embodiment> In the first and second embodiments, the processing system 10 was an accounting system having an accounting device and a server as shown in FIG. 2. Unlike the first and second embodiments, the processing system 10 of this embodiment has a server that communicates with a terminal device 20 operated by a customer as shown in FIG. 9. The processing system 10 outputs the recognition result and accepts input of corrections to the recognition result via the terminal device 20. The terminal device 20 may be a customer terminal such as a smartphone, tablet terminal, smart watch, mobile phone, or PC (personal computer), or may be a dedicated terminal installed in a store, or may be something else.
[0072] An example of a functional block diagram of the processing system 10 of this embodiment is shown in FIG. 4, similarly to the first and second embodiments.
[0073] In this embodiment, a camera is installed in a store at a position and orientation that allows it to capture an image of a customer removing a product from a product shelf. The camera may be installed on a product shelf, on a ceiling, on a floor, on a wall, or in another location.
[0074] In addition, the number of cameras that capture the customer taking out a product from one product shelf may be one or more. When the customer taking out a product from one product shelf is captured by multiple cameras, it is preferable that the multiple cameras are installed so as to capture the customer taking out the product from the product shelf from different positions and directions.
[0075] In addition, a camera may be installed on each product shelf, or on every multiple product shelves, or on every tier of a product shelf, or on every multiple tiers of a product shelf.
[0076] The camera may capture moving images continuously (e.g., during business hours), or it may continuously capture still images at time intervals longer than the frame interval of the moving images, or it may perform these captures only while a motion sensor or the like detects a person in a specified position (such as in front of a product shelf).
[0077] Here, an example of camera installation is shown. Note that the camera installation example described here is merely an example, and is not limited to this. In the example shown in Fig. 10, two cameras 2 are installed for each product shelf 1. Fig. 11 is a diagram extracting the frame 4 from Fig. 10. Each of the two components that make up the frame 4 is provided with a camera 2 and lighting (not shown).
[0078] The light emission surface of the lighting extends in one direction, and has a light-emitting unit and a cover that covers the light-emitting unit. The lighting mainly emits light in a direction perpendicular to the extension direction of the light-emitting surface. The light-emitting unit has a light-emitting element such as an LED, and emits light in a direction that is not covered by the cover. When the light-emitting element is an LED, multiple LEDs are lined up in the direction in which the lighting extends (up and down in the figure).
[0079] Camera 2 is provided at one end of the linearly extending frame 4 component, and its imaging range is the direction in which the illumination light is emitted. For example, in the frame 4 component on the left side of Fig. 11, camera 2's imaging range is downward and diagonally downward to the right. Also, in the frame 4 component on the right side of Fig. 11, camera 2's imaging range is upward and diagonally upward to the left.
[0080] As shown in FIG. 10, the frame 4 is attached to the front frame (or the front of both side walls) of the product shelf 1 that constitutes the product placement space. One of the components of the frame 4 is attached to one of the front frames with the camera 2 facing downward. The other of the components of the frame 4 is attached to the other of the front frames with the camera 2 facing upward. The camera 2 attached to one of the components of the frame 4 takes pictures above and diagonally upward so that the opening of the product shelf 1 is included in the shooting range. Meanwhile, the camera 2 attached to the other of the components of the frame 4 takes pictures below and diagonally downward so that the opening of the product shelf 1 is included in the shooting range. With this configuration, the two cameras 2 can shoot the entire range of the opening of the product shelf 1.
[0081] 4 acquires a recognition-processed image generated by such a camera. The recognition-processed image may be input to the processing system 10 by real-time processing or by batch processing. Which processing is used can be determined depending on, for example, how the recognition result is to be used.
[0082] The configurations of the recognition unit 12 and the registration unit 13 are the same as those in the first embodiment. The registration unit 13 only needs to link and register at least the recognition result and the recognition processing image on which the recognition result is based, and does not necessarily need to register information such as the product name and unit price acquired from the store server. Depending on the usage of the recognition result, it is possible to select whether or not to register the information acquired from the store server.
[0083] The output unit 14 outputs the recognition result to the customer via the terminal device 20. For example, similar to the first and second embodiments, the output unit 14 causes the terminal device 20 to display a screen displaying a list of the recognition results as shown in Fig. 6. Then, the correction accepting unit 16 accepts an input for correcting the recognition result via the terminal device 20. A specific example will be described below.
[0084] "Example 1" The processing system 10 recognizes the product picked up by the customer using the above configuration, and identifies the customer who picked up the product using any means. The processing system 10 then registers recognized product information (recognition result) as shown in Fig. 5, linked to the customer identification information of the customer. The means for identifying the customer may be realized, for example, by face recognition processing based on a face image of the customer captured by a camera installed in the store, or may be realized by other means.
[0085] Then, the output unit 14 outputs the recognition result via the terminal device 20 of each customer. Also, the correction receiving unit 16 receives input for correcting the recognition result via the terminal device 20 of each customer. For example, each customer accesses the processing system 10 via a predetermined application installed in the terminal device 20, and logs in using his / her own customer identification information. Then, the processing system 10 identifies the terminal device 20 of each customer based on the login information, and outputs the recognition result associated with each customer or receives input for correcting the recognition result via the terminal device 20 of each identified customer.
[0086] "Example 2" The processing system 10 recognizes the product picked up by the customer using the above configuration, and identifies the customer who picked up the product using any means. The processing system 10 then registers recognized product information (recognition result) as shown in Fig. 5, linked to the customer identification information of the customer. The means for identifying the customer may be realized, for example, by face recognition processing based on a face image of the customer captured by a camera installed in the store, or may be realized by other means.
[0087] The output unit 14 outputs the recognition result via a terminal device 20 installed in the store. The correction receiving unit 16 receives an input for correcting the recognition result via the terminal device 20 installed in the store. The terminal device 20 installed in the store may be an accounting device such as a POS (point of sale) register, or may be something else.
[0088] For example, when making a payment, a customer inputs his / her customer identification information into a terminal device 20 installed in a store. For example, the customer may input the information by photographing his / her face. In this case, the customer identification information is identified by face recognition processing based on the photographed face image of the customer. Alternatively, the customer may input the information by enabling communication between a reader that performs short-range wireless communication and a device that stores the customer identification information (smartphone, smartwatch, tablet terminal, mobile phone, IC card, etc.). Alternatively, the customer may input the customer identification information via an input device such as a touch panel, microphone, keyboard, mouse, etc.
[0089] When the processing system 10 acquires customer identification information from a terminal device 20 installed in a store, the processing system 10 transmits the recognition result associated with the customer identification information to the terminal device 20 and causes it to be displayed. The processing system 10 also accepts an input to correct the recognition result associated with the customer identification information via the terminal device 20.
[0090] "Specific example 3" When the processing system 10 recognizes a product picked up by a customer using the above-mentioned configuration, it registers recognized product information (recognition result) as shown in Figure 5, linking it to the facial image of the customer who picked up the product and / or features extracted from the facial image.
[0091] The output unit 14 then outputs the recognition result via a terminal device 20 installed in the store. The correction receiving unit 16 also receives input for correcting the recognition result via the terminal device 20 installed in the store. The terminal device 20 installed in the store may be an accounting device such as a POS register, or may be something else.
[0092] For example, when a customer performs a checkout process, the customer has the terminal device 20 installed in the store take a picture of his / her face. When the processing system 10 acquires a facial image of the customer from the terminal device 20 installed in the store, the processing system 10 transmits the acquired facial image or a recognition result linked to the feature amount extracted from the facial image to the terminal device 20 and displays it. The processing system 10 also accepts an input via the terminal device 20 to correct the recognition result linked to the customer identification information.
[0093] Returning to FIG. 4, the configurations of the storage unit 15, the correction unit 17 and the learning unit 18 are similar to those in the first and second embodiments.
[0094] According to the processing system 10 of this embodiment, the same action and effect as the first and second embodiments is realized. Also, according to the processing system 10 of this embodiment, it is possible to realize the generation of a recognition processing image, the output of the recognition result, and the input for correcting the recognition result, by a method different from that of the first and second embodiments. As a result, the application scenario of the processing system 10 is expanded, which is preferable.
[0095] In this specification, "acquisition" includes at least one of the following: "the device itself goes to retrieve data stored in another device or storage medium (active acquisition)" based on user input or based on program instructions, such as receiving data by making a request or inquiry to another device, or accessing another device or storage medium and reading it, and "the device itself inputs data output from another device (passive acquisition)" based on user input or based on program instructions, such as receiving data that is distributed (or transmitted, push notification, etc.), and selecting and acquiring data from among the received data or information, and "editing data (converting it to text, rearranging data, extracting some data, changing the file format, etc.) to generate new data and acquiring the new data."
[0096] Although the present invention has been described above with reference to the embodiments (and examples), the present invention is not limited to the above-mentioned embodiments (and examples). Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.
[0097] A part or all of the above-described embodiments can be described as follows, but is not limited to the following. 1. An image acquisition means for acquiring a recognition processing image including a recognition target product; A recognition means for recognizing a product in the recognition processing image based on an estimation model generated by machine learning; a registration means for registering the result of the recognition in recognized product information; an output means for outputting the result of the recognition; a correction receiving means for receiving an input for correcting the result of the recognition; a correction means for changing the recognition result registered in the recognized product information to a corrected recognition result, and storing correction information linking the corrected recognition result with the recognition processing image in a storage means; a learning means for updating the estimation model by re-learning using the recognition processing images stored as the correction information when the number of the recognition processing images stored as the correction information exceeds a predetermined value; A processing system having: 2. The output means displays a list of the recognition results registered in the recognized product information; The processing system according to claim 1, wherein the correction accepting means accepts an input to specify one of the multiple recognition results displayed in a list, and an input to correct the specified recognition result. 3. A processing system according to 1 or 2, wherein the correction receiving means receives input of correct product identification information as input for correcting the recognition result. 4. The processing system according to claim 3, wherein the correction accepting means accepts input of correct product identification information via a code reader. 5. The correction receiving means receives input of correct product identification information via a code reader as an input for correcting the recognition result, The image acquisition means further acquires a correction image showing a state of an operation of causing the code reader to read the product identification information attached to the product, The correction means identifies an object to be corrected among the recognition results registered in the recognized product information based on the correction image. 6. The recognition means recognizes the product in the corrected image based on the estimation model; The processing system described in 5, wherein the correction means identifies, among the recognition results registered in the recognized product information, those that match the recognition results of the correction image as those to be corrected. 7. The processing system described in 5, wherein the correction means identifies the object to be corrected based on the similarity between the external features of the product in the correction image and the external features of the product in the recognition processing image. 8. The computer: Acquire a recognition processing image including the product to be recognized; Recognizing the product in the recognition processing image based on an estimated model generated by machine learning; The result of the recognition is registered in the recognized product information; Outputting the results of the recognition; Accepting input to correct the recognition result; changing the recognition result registered in the recognized product information to the corrected recognition result, and storing correction information linking the corrected recognition result with the recognition processing image in a storage means; A processing method in which, when the number of the recognition processing images stored as the correction information exceeds a predetermined value, re-learning is performed using the recognition processing images stored as the correction information to update the estimation model. 9. Computer An image acquisition means for acquiring a recognition processing image including a recognition target product; A recognition means for recognizing a product in the recognition processing image based on an estimation model generated by machine learning; a registration means for registering the result of the recognition in recognized product information; an output means for outputting the result of the recognition; a correction receiving means for receiving an input for correcting the result of the recognition; a correction means for changing the recognition result registered in the recognized product information to a corrected recognition result, and storing correction information linking the corrected recognition result with the recognition processing image in a storage means; a learning means for updating the estimation model by re-learning using the recognition processing images stored as the correction information when the number of the recognition processing images stored as the correction information exceeds a predetermined value; A program that functions as a
Claims
1. An image acquisition means for acquiring an image for recognition processing including an object; A recognition means for recognizing an object in the recognition processing image based on an estimation model generated by machine learning; a correction receiving means for receiving an input for correcting the result of the recognition; a correction means for storing correction information in a storage means, the correction information linking the result of the recognition after the correction with the recognition processing image; a learning means for counting the number of the recognition processing images stored as the correction information for each object, and when the number exceeds a predetermined value, performing re-learning using the recognition processing images stored as the correction information to update the estimation model; A processing system having:
2. a registration means for registering the result of the recognition as recognition information; an output means for outputting the result of the recognition; having 2. The processing system according to claim 1, wherein the correction means changes the recognition result registered in the recognition information to the corrected recognition result.
3. the output means displays a list of the plurality of recognition results registered in the recognition information; 3. The processing system according to claim 2, wherein the correction accepting means accepts an input for designating one of the plurality of recognition results displayed in a list, and an input for correcting the designated recognition result.
4. 4. The processing system according to claim 1, wherein the correction receiving means receives an input of correct object identification information as an input for correcting the recognition result.
5. 5. The processing system according to claim 4, wherein the correction accepting means accepts input of correct object identification information via a code reader.
6. the correction receiving means receives an input of correct object identification information via a code reader as an input for correcting the recognition result; The image acquisition means further acquires a correction image showing a state of an operation of causing the code reader to read the object identification information attached to the object, 4. The processing system according to claim 3, wherein the correction means specifies an object to be corrected from the recognition result registered in the recognition information based on the correction image.
7. The recognition means recognizes an object in the corrected image based on the estimated model; 7. The processing system according to claim 6, wherein the correction means specifies, as an object to be corrected, a result of the recognition registered in the recognition information that coincides with a result of the recognition of the correction image.
8. 7. The processing system according to claim 6, wherein the correction means specifies an object to be corrected based on a similarity between an amount of characteristic of an appearance of an object in the correction image and an amount of characteristic of an appearance of an object in the recognition processing image.
9. The computer Obtaining a recognition processing image including an object; Recognizing an object in the recognition processing image based on an estimated model generated by machine learning; Accepting input to correct the recognition result; storing correction information linking the corrected recognition result and the recognition processing image in a storage means; A processing method in which the number of the recognition processing images stored as the correction information is counted for each object, and when the number exceeds a predetermined value, the estimation model is updated by re-learning using the recognition processing images stored as the correction information.
10. Computer, An image acquisition means for acquiring an image for recognition processing including an object; A recognition means for recognizing an object in the recognition processing image based on an estimation model generated by machine learning; a correction receiving means for receiving an input for correcting the result of the recognition; a correction means for storing in a storage means correction information linking the corrected recognition result and the recognition processing image; a learning means for counting the number of the recognition processing images stored as the correction information for each object, and when the number exceeds a predetermined value, performing re-learning using the recognition processing images stored as the correction information to update the estimation model; A program that functions as a
Citation Information
Patent Citations
Commodity recognition device and commodity recognition program
JP2015187870A
Commodity register
JP2016062545A
Learning device, learning method and program
JP2016143354A
Counting device, accounting system, learning device, and control method
JP2019200533A
Information processing system
JP2020061044A
