Data processor, method for processing data, and data processing program

Abstract

To provide a data processor which increases the efficiency of re-learning or additional learning to generate dictionary data used in recognition processing.SOLUTION: The data processor according to an embodiment includes an interface and a processor. The interface inputs predetermined registration candidate image data regarding a predetermined item. The processor determines whether registering the predetermined registration candidate image data satisfies the standard of increasing the recognition rate of the predetermined item.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] An embodiment of the present invention relates to a data processing device, a data processing method, and a data processing program. [Background technology]

[0002] Learning-based item recognition processing can be more robust to variations in the appearance of the item to be recognized than rule-based recognition processing. However, even for the same item, if the appearance of the item contained in the registered image data is significantly different from that of the item contained in the actual photographed image data, the recognition performance may be degraded. To prevent the recognition performance from degrading, it is effective to update the dictionary data by re-learning or additional learning of the photographed image data of the target item. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-81081 [Patent Document 2] Patent Publication No. 2021-128695 Summary of the Invention [Problem to be solved by the invention]

[0004] Re-learning or additional learning requires not only photographing the target item, but also a teaching process based on the photographed image data obtained by photographing. When new items are added one after another, the burden of the photographing and teaching process is not light. In the photographing and teaching process, the support of a person with specialized knowledge may be required to check whether an item with a similar appearance to the target item has already been learned, or whether new image data needs to be added to ensure recognition performance.

[0005] In addition, due to failure to recognize an item, the robot may misalign the position at which the item is grasped. Such misalignment of the item's grasp position may lead to damage to the item. In an attempt to prevent such problems, the number of teaching steps may increase excessively.

[0006] In order to maintain the recognition accuracy, re-learning or additional learning is necessary, but it imposes a certain load. For this reason, there is a demand to exclude images that are unlikely to contribute to improving the recognition rate from the targets of re-learning or additional learning.

[0007] An object of the present invention is to provide a data processing device, a data processing method, and a data processing program that improve the efficiency of re-learning or additional learning for generating dictionary data used in recognition processing. [Means for solving the problem]

[0008] A data processing device according to an embodiment includes an interface and a processor. The interface inputs predetermined registration candidate image data related to a predetermined item. The processor determines whether or not registering the predetermined registration candidate image data satisfies a recognition rate improvement criterion for the predetermined item. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an application example of a control system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a data processing device according to the embodiment. [Diagram 3] FIG. 3 is a block diagram illustrating an example of a control device according to the embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of an image acquisition process by the data processing device according to the embodiment. [Diagram 5] FIG. 5 is a flowchart showing an example of an image registration process by the data processing device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Each embodiment will be described below with reference to the drawings.

[0011] [composition] FIG. 1 is a diagram illustrating an application example of a control system according to an embodiment. The control system 1 comprises a data processing device 2 and a control device 3 . The data processing device 2 provides the control device 3 with dictionary data and the like required for controlling the picking robot 62 and the like. The data processing device 2 can be configured with one or more computers, that is, a processor, a memory, an interface, and the like.

[0012] For example, the data processing device 2 determines whether or not registering the photographed image data, which is the registration candidate image data, contributes to improving the recognition rate of the item, and if so, registers it as data used for learning, and provides the control device 3 with dictionary data generated based on the learning of the data used for learning. For example, in the learning process, the data processing device 2 generates a parameter group of the neural network from the input (photographed image data output from the photographing device 53) and the correct answer (teaching data = item area in the photographed image data). In this case, the dictionary data includes the generated parameter group. One parameter group is used generically for a certain item group. In other words, when it is necessary to recognize multiple types of items, the registration candidate image data will be images of multiple types of items, but the dictionary data including one parameter group makes it possible to recognize these multiple types of items. Note that the dictionary data may be data including an image data group used in recognition processing such as pattern matching.

[0013] The control device 3 controls the picking robot 62 and the like arranged at the station. The control device 3 receives and stores dictionary data provided from the data processing device 2. In the inference (recognition) process, the control device 3 generates an inference result (recognition result = item area in the photographed image data) from the input (photographed image data output from the photographing device 54) and the dictionary data. That is, the control device 3 recognizes an item from the photographed image data, and outputs a control signal to the picking robot 62 to pick up the item based on the recognition result. The control device 3 can be configured with one or more computers, i.e., a processor, a memory, an interface, and the like.

[0014] In the station, a conveyor 4 is disposed along which items 8 are transported. In a work area 51 along the conveyor 4, a worker 61 is disposed, and in another work area 52, a picking robot 62 is disposed. Into the work areas 51 and 52, a container (TOTE) 7 is carried based on an order list from a higher-level server. In the container 7, one or more items 8 specified based on the order list are stored.

[0015] For example, the order list includes additional information about the target items. For example, the additional information includes item type information, item size information, information about the number of items in the container, information about the date and time of entry or exit, and information about the appearance of the items. In addition, item information such as a two-dimensional code is attached to the items 8. The item information includes item type information and item identification information.

[0016] The camera 53 is disposed corresponding to the work area 51. The camera 53 is connected to the data processing device 2. The camera 53 captures images of the work area 51. Image data of a moving image or a still image output from the camera 53 includes an image of the container 7 that is the subject of work in the work area 51. This image also includes an image of one or more items 8 stored in the container 7.

[0017] Similarly, the camera 54 is disposed corresponding to the work area 52. The camera 54 is connected to the control device 3. The camera 54 captures images of the work area 52. Image data of a moving image or a still image output from the camera 54 includes an image of the container 7 that is the subject of work in the work area 52. This image also includes images of one or more items 8 stored in the container 7.

[0018] FIG. 2 is a block diagram illustrating an example of a data processing device according to the embodiment. As shown in FIG. 2, the data processing device 2 includes a processor 21, a memory 22, and an interface .

[0019] The processor 21 is a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), or the like. The processor 21 operates based on a program stored in a memory 22. The processor 21 includes an input unit 211, a determination unit 213, a registration unit 214, and a recognition unit 215. The processor 21 executes the operation program stored in the memory 22 to realize the functions of each unit.

[0020] The memory 22 is a non-volatile memory, and stores an operation program for the processor 21. The memory 22 also includes a management information area 221, a learning data area 222, and a dictionary data area 223.

[0021] The management information area 221 stores information on items provided from the outside. For example, the management information area 221 stores an order list transmitted from a higher-level server, registration candidate image data output from the photographing device 53, and item information transmitted from the reader 9. For example, the order list includes auxiliary information, and the auxiliary information includes item type information. Based on the timing of receiving the order list, the auxiliary information included in the order list may be associated with the input registration candidate image data. The photographing device 53 outputs image data at a predetermined timing, and the management information area 221 stores the image data output from the photographing device 53. In addition, the item information includes item type information. Based on the timing of receiving the item information, the auxiliary information included in the item information may be associated with the input registration candidate image data. In this way, the auxiliary information being associated with the input registration candidate image data corresponds to the item type information of the item included in the registration candidate image data being associated with the input registration candidate image data.

[0022] In the learning data area 222, image data that satisfies the recognition rate improvement standard among the registration candidate image data output from the photographing device 53 and temporarily stored in the management information area 221 is registered as learning data to be used for learning. For example, the processor 21 determines whether or not registering the predetermined registration candidate image data related to a predetermined item satisfies the recognition rate improvement standard of the predetermined item, and if it is determined that it satisfies the standard, moves or copies the predetermined registration candidate image data stored in the management information area 221 to the learning data area 222 to register the predetermined registration candidate image data. Alternatively, the processor 21 may assign a flag to the predetermined registration candidate image data stored in the management information area 221 and register the predetermined registration candidate image data. Note that instead of registering the learning data in the data processing device 2, the learning data may be registered in an external server or the like. Also, as described above, the registration candidate image data is associated with auxiliary information or the like. Based on the auxiliary information, it is determined whether or not registering the registration candidate image data of the same type of item as the item to be recognized by the dictionary data satisfies the recognition rate improvement standard of this item, and if the recognition rate improvement standard is satisfied, the registration candidate image data is registered. The registration process of the registration candidate image data will be described in detail later.

[0023] In the learning process, dictionary data including a group of parameters generated from the input (photographed image data output from the photographing device 53) and the correct answer (teaching data = item area in the photographed image data) is registered in the dictionary data area 223. The input described above is image data that satisfies the recognition rate improvement standard and is registered as data to be used for learning, among the candidate image data to be registered that is output from the photographing device 53. For example, the learner visually determines the area of ​​a specific item included in the image data registered as data to be used for learning, and specifies the area, thereby generating the teaching data. Alternatively, the processor 21 detects the area of ​​a specific item included in the image data registered as data to be used for learning, and the learner corrects the area as necessary, thereby generating the teaching data.

[0024] The dictionary data inherits the auxiliary information from the registration candidate image data, that is, the dictionary data is associated with auxiliary information related to items included in the registration candidate image data.

[0025] Furthermore, the dictionary data in the dictionary data area 223 is provided to the control device 3. The control device 3 generates an inference result (recognition result=item area in the captured image data) from the input (captured image data output from the photographing device 54) and the dictionary data. That is, the control device 3 recognizes the item to be picked from the captured image data, and outputs a control signal to the picking robot 62 to pick up the item based on the recognition result.

[0026] The interface 23 inputs photographed image data output from the photographing device 53. The photographing device 53 photographs the work area 51 and outputs the obtained photographed image data. The work area 51 is an area where items are handled by the worker 61. The photographed image data output from the photographing device 53 is the above-mentioned registration candidate image data. In addition, the interface 33 transmits the dictionary data in the dictionary data area 223 to the control device 3.

[0027] Furthermore, the interface 23 inputs the item information output from the reader 9. For example, the reader 9 is a two-dimensional code reader that reads the two-dimensional code of the item information attached to the item 8 and outputs the read item information.

[0028] FIG. 3 is a block diagram illustrating an example of a control device according to the embodiment. As shown in FIG. 3, the control device 3 includes a processor 31, a memory 32, and an interface 33.

[0029] The processor 31 is a CPU, an MPU, a DSP, or the like. The processor 31 operates based on a program stored in the memory 32. The processor 31 includes an input unit 311, a recognition unit 312, and a control unit 313. The processor 31 executes the operation program stored in the memory 32 to realize the functions of each unit.

[0030] The memory 32 is a non-volatile memory, and stores an operation program for the processor 31. The memory 32 also includes a management information area 321 and a dictionary data area 322. Information on items provided from the outside is registered in the management information area 321. For example, an order list transmitted from a higher-level server and photographed image data output from the photographing device 54 are stored in the management information area 321. Dictionary data provided from the data processing device 2 is registered in the dictionary data area 322.

[0031] The interface 33 inputs photographed image data output from the photographing device 54. The photographing device 54 photographs the work area 52 and outputs the obtained photographed image data. The work area 52 is an area where items are processed by the robot 62. The photographed image data output from the photographing device 54 is data provided for item recognition processing by the processor 31. The interface 33 also receives dictionary data transmitted from the data processing device 2.

[0032] The input unit 311 inputs photographed image data output from the photographing device 54 via the interface 33 .

[0033] The recognition unit 312 generates an inference result (recognition result=item area in the captured image data) from the input (captured image data output from the photographing device 54) and dictionary data including a group of parameters in the dictionary data area 322. That is, the recognition unit 312 recognizes an item from the captured image data. When the dictionary data in the dictionary data area 322 is a group of image data used in recognition processing such as pattern matching, the recognition unit 312 may detect an outline likely to be an item from the captured image data, detect an outline candidate having the highest similarity to the detected outline from multiple outline candidates included in the dictionary data, and identify the item based on item-related information associated with the detected outline candidate.

[0034] The control unit 313 outputs a control signal for controlling the picking of the item based on the result of the item recognition. The picking robot 62 picks the item based on the control signal.

[0035] [Operation] Re-learning is necessary when adding a new type of item or when the recognition rate of a specific type of item is poor, but the data processing device 2 according to the embodiment judges these cases as cases in which registering predetermined registration candidate image data satisfies the recognition rate improvement standard for the predetermined item, and supports efficient re-learning. The operation of the data processing device 2 will be described below.

[0036] FIG. 4 is a flowchart showing an example of an image acquisition process by the data processing device according to the embodiment. The interface 23 receives the order list transmitted from the upper server, and the memory 22 stores the order list in the management information area 221. The order list includes additional information related to the items to be put into or taken out of stock, and the additional information includes item type information and the like.

[0037] Items 8 based on the order list are brought into the work area. One or more items 8 based on the order list are stored in the container 7. Item information is attached to the items 8.

[0038] For example, a worker 61 assigned to the work area 51 picks an item 8, and holds a reader 9 over the item information attached to the item 8 to have the reader 9 read the item information. The item information includes item type information, etc. The worker 61 releases the picked item 8 onto the conveyor 4.

[0039] The photographing device 53 photographs the work area 51 where the worker 61 works, and outputs photographed image data obtained by photographing.

[0040] The interface 23 receives the photographed image data output from the photographing device 53 as registration candidate image data, and also receives the item information output from the reader 9.

[0041] For example, the interface 23 may input a part of the photographed image data from the photographing device 53 as registration candidate image data based on the timing of receiving the order list. That is, the interface 23 uses the timing of receiving the order list as a trigger (ST101, YES) and inputs the photographed image data output from the photographing device 53 around the time as registration candidate image data (ST102).

[0042] The recognition unit 215 recognizes item areas, etc. from the registration candidate image data in order to support the learner in specifying teaching data (ST103), and stores predetermined registration candidate image data and the recognition results of the item areas, etc. in the management information area 221 of the storage unit 22 (ST104). Note that the recognition of the item areas is not essential, and the learner may visually distinguish the item areas from the image data and specify the item areas.

[0043] For example, the memory 22 stores the predetermined registration candidate image data in the management information area 221 in association with the auxiliary information included in the order list. That is, the predetermined registration candidate image data associated with the predetermined auxiliary information related to the predetermined item is saved by being triggered by the timing of receiving an order list including the predetermined auxiliary information related to the predetermined item. In other words, the interface 23 inputs the predetermined registration candidate image data and the predetermined auxiliary information related to the predetermined item.

[0044] Alternatively, the interface 23 may input a part of the photographed image data from the photographing device 53 as the registration candidate image data based on the timing of receiving the item information output from the reader 9. That is, the interface 23 uses the timing of receiving the item information as a trigger (ST101, YES) and inputs the photographed image data output from the photographing device 53 around the time as the registration candidate image data (ST102). In this case, the predetermined registration candidate image data associated with the item information is saved.

[0045] Alternatively, the interface 23 may input a part of the photographed image data from the photographing device 53 as the registration candidate image data based on the timing of receiving a detection signal that detects that the container 7 has arrived at a predetermined area. That is, the interface 23 uses the timing of receiving the detection signal as a trigger (ST101, YES) and inputs the photographed image data output from the photographing device 53 around the time as the registration candidate image data (ST102). For example, a plurality of sensors that detect the container 7 may be arranged around the station, and the detection signals from these sensors may be used as the trigger.

[0046] Alternatively, the interface 23 may input a part of the captured image data from the photographing device 53 as the registration candidate image data based on the timing of receiving a detection signal that detects a predetermined movement of the worker 61. That is, the interface 23 uses the timing of receiving the detection signal as a trigger (ST101, YES) and inputs the captured image data output from the photographing device 53 around the time as the registration candidate image data (ST102). For example, a plurality of cameras may be placed around the station, and a detection signal that detects a predetermined movement based on the analysis results of the image data output from these cameras may be used as the trigger.

[0047] By repeating the above operations, registration candidate image data obtained from the photographed image data in the work area 51 is stored in the management information area 221.

[0048] Based on the above-mentioned trigger, the photographing device 53 may start photographing and output image data, or may start outputting image data during photographing.

[0049] FIG. 5 is a flowchart showing an example of an image registration process by the data processing device according to the embodiment. The input unit 211 inputs data stored in the management information area 221, for example, predetermined registration candidate image data associated with predetermined auxiliary information related to a predetermined item, the recognition result of an item area, and the like (ST201).

[0050] The determination unit 213 determines whether or not the recognition rate improvement standard is satisfied based on the registration candidate image data associated with the predetermined auxiliary information and the dictionary data associated with the predetermined auxiliary information for recognizing a predetermined item by image processing. In other words, the determination unit 213 determines whether or not the recognition rate improvement standard is satisfied based on the recognition result using the registration candidate image data associated with the predetermined auxiliary information and the dictionary data associated with the predetermined auxiliary information.

[0051] For example, the determination unit 213 determines whether or not registering the predetermined registration candidate image data of the same type of item as the item to be recognized by the dictionary data satisfies the recognition rate improvement standard for the predetermined item based on the predetermined auxiliary information (ST202). In other words, the determination unit 213 determines whether or not registering the predetermined registration candidate image data satisfies the recognition rate improvement standard for the predetermined item.

[0052] Furthermore, the determination unit 213 may determine whether or not registering the predetermined registration candidate image data of the same type of item as the item to be recognized by the dictionary data satisfies the recognition rate improvement standard for the predetermined item based on the size information of the predetermined item included in the predetermined auxiliary information. That is, the determination unit 213 determines that the recognition has failed if the sizes of the predetermined items are significantly different. In such a case, it is worth teaching and learning. Therefore, the determination unit 213 determines that the recognition rate improvement standard is satisfied in such a case. Conversely, the determination unit 213 determines that the recognition has succeeded if the difference in the sizes of the predetermined items is less than the reference value. In such a case, it is not worth teaching and learning. Therefore, the determination unit 213 determines that the recognition rate improvement standard is not satisfied in such a case, and excludes the data from the learning data used for learning.

[0053] Furthermore, the determination unit 213 may determine whether or not registering a predetermined registration candidate image data of an item of the same type as an item to be recognized by the dictionary data satisfies a recognition rate improvement standard for the predetermined item based on the appearance information of the predetermined item (for example, an image of the item) included in the predetermined auxiliary information. In other words, the determination unit 213 determines that recognition has failed if the appearance of the predetermined item is significantly different. In such a case, it is worth teaching and learning. Therefore, the determination unit 213 determines that the recognition rate improvement standard is satisfied in such a case. Conversely, the determination unit 213 determines that recognition has succeeded if the difference in appearance of the predetermined item is less than a reference value. In such a case, it is not worth teaching and learning. Therefore, the determination unit 213 determines that the recognition rate improvement standard is not satisfied in such a case, and excludes the data from the learning data used for learning.

[0054] If the recognition rate improvement standard is met (ST203, YES), the registration unit 214 registers the predetermined registration candidate image data in the training data area 222 or the like as training data to be used for training (ST204). If the recognition rate improvement standard is not met (ST203, NO), the registration unit 214 excludes the predetermined registration candidate image data from the training data to be used for training and does not register the predetermined registration candidate image data.

[0055] The processor 21 generates parameters based on predetermined registration candidate image data and dictionary data, and if the recognition rate improvement standard is satisfied, updates the dictionary data based on the generated parameters.

[0056] The determination of whether or not the recognition rate improvement standard is satisfied will be described in detail. The determination unit 213 determines, based on the predetermined auxiliary information, whether or not registering predetermined registration candidate image data of an item of the same type as an item to be recognized by the dictionary data satisfies a recognition rate improvement standard.

[0057] For example, when the consistency of the recognition result of the item region satisfies a consistency criterion, the determination unit 213 determines that the recognition rate improvement criterion is satisfied.

[0058] The determination unit 213 determines that the matching criterion is met when the recognition result of the item area based on the registration candidate image data in the input state and the recognition result of the item area based on the data obtained by processing the input registration candidate image data are the same or the difference between these recognition results is less than a reference value. Processing includes, for example, left-right inversion, up-down inversion, or 90-degree rotation.

[0059] Alternatively, the determination unit 213 determines that the recognition rate improvement criterion is met when the consistency between the recognition result of the item area using the registration candidate image data and the dictionary data and the recognition result of the item area using the data obtained by processing the registration candidate image data and the dictionary data satisfies the consistency criterion. For example, the determination unit 213 determines that the recognition rate improvement criterion is met when the consistency between the outline, area, or pattern of a predetermined item recognized from the registration candidate image data and the dictionary data and the outline, area, or pattern of a predetermined item recognized from the data obtained by processing the registration candidate image data and the dictionary data satisfies the consistency criterion.

[0060] A supplementary explanation of an example of processing the registration candidate image data is provided. For example, the determination unit 213 determines that the matching criterion and the recognition rate improvement criterion are met when the recognition result of an item area using the specified registration candidate image data and dictionary data in the input state is identical to the recognition result of an item area using the specified registration candidate image data and dictionary data in the state flipped left / right, flipped up / down, or rotated 90 degrees, or when the difference between these recognition results is less than a reference value. In other words, the determination unit 213 determines that the matching criterion and the recognition rate improvement criterion are met when an identical area or a similar area with a difference less than a reference value is detected from either state.

[0061] Provided that the matching criteria are satisfied, the predetermined registration candidate image data to be registered is image data that contributes greatly to improving the recognition rate.

[0062] Furthermore, the determination unit 213 determines whether or not registering the predetermined registration candidate image data satisfies a recognition rate improvement criterion for the predetermined item based on the recognition history of the predetermined item using the dictionary data. For example, the determination unit 213 determines that the recognition rate improvement criterion is satisfied when the recognition history includes one recognition failure. The determination unit 213 may change the criterion and determine that the recognition rate improvement criterion is satisfied when the recognition history includes multiple recognition failures.

[0063] As described above, if recognition fails even once, it is possible to pursue higher recognition accuracy and promote the registration of registration candidate image data, or conversely, if recognition fails only once, it is possible not to promote the registration of registration candidate image data so as not to increase the burden on the teaching process.

[0064] Alternatively, the determination unit 213 may determine that the recognition rate improvement criterion is satisfied based on a recognition failure rate or a recognition success rate detected from the number of recognition failures and recognition successes included in the recognition history. If the recognition failure rate is higher than the criterion or the recognition success rate is lower than the criterion, the determination unit 213 may determine that the recognition rate improvement criterion is satisfied.

[0065] If the control device 3 fails to recognize an item based on the dictionary data registered in the dictionary data, it registers a recognition failure in the recognition history associated with the dictionary data, and if the recognition is successful, it registers a recognition success in the recognition history associated with the dictionary data. The control device 3 transmits dictionary data including the dictionary data associated with the recognition history to the data processing device 2. The data processing device 2 receives the dictionary data with the recognition history updated by the control device 3 and stores it in the dictionary data area 223.

[0066] Furthermore, the determination unit 213 may combine the above-mentioned consistency with the recognition history to determine whether or not registering predetermined registration candidate image data satisfies a criterion for improving the recognition rate of a predetermined item.

[0067] If the consistency does not satisfy the consistency criterion, the determination unit 213 determines that the recognition rate improvement criterion is not satisfied. In other words, if the registration candidate image data is significantly different from the dictionary data even though it is the same type of item, it is difficult to expect an improvement in the recognition rate. Therefore, it is determined that the recognition rate improvement criterion is not satisfied.

[0068] Furthermore, when the consistency satisfies the consistency criterion and the recognition history indicates a high recognition success rate, the determination unit 213 determines that the recognition rate improvement criterion is not met. Since the recognition success rate is high to begin with, it is difficult to expect a further improvement in the recognition rate even if registration candidate image data is added to the dictionary data. Therefore, it is determined that the recognition rate improvement criterion is not met.

[0069] Furthermore, when the consistency satisfies the consistency criterion and the recognition history indicates a low recognition success rate, the determination unit 213 determines that the recognition rate improvement criterion is met. Since the recognition success rate is low, adding registration candidate image data to the dictionary data is expected to improve the recognition rate. Therefore, it is determined that the recognition rate improvement criterion is met.

[0070] Furthermore, processor 22 stores a plurality of predetermined registration candidate image data of a predetermined item in learning data area 222. Determination unit 213 may determine whether or not the determination result for the plurality of predetermined registration candidate image data satisfies a predetermined criterion based on the plurality of predetermined registration candidate image data of a predetermined item and dictionary data, and determine whether or not the recognition rate improvement criterion for the predetermined item is satisfied based on whether or not the predetermined criterion is satisfied.

[0071] For example, assume that 10 patterns of registration candidate image data for a predetermined item are stored in learning data area 222. Determination unit 213 determines that the predetermined criterion is met when the area of ​​the predetermined item can be recognized from all 10 patterns. Determination unit 213 determines that re-learning is unnecessary when the predetermined criterion is met because the recognition rate is high, and determines that the recognition rate improvement criterion for the predetermined item is not met based on the determination that the predetermined criterion is met. Determination unit 213 also determines that the predetermined criterion is not met when the area of ​​the predetermined item cannot be recognized from all 10 patterns, and determines that the recognition rate improvement criterion for the predetermined item is met based on the determination that the predetermined criterion is not met.

[0072] Alternatively, the determination unit 213 determines that the predetermined criterion is met when the area of ​​the predetermined item can be recognized from eight or more patterns (80% or more). When the predetermined criterion is met, the determination unit 213 determines that re-learning is unnecessary because the recognition rate is high, and determines that the recognition rate improvement criterion for the predetermined item is not met based on the determination that the predetermined criterion is met. Furthermore, the determination unit 213 determines that the predetermined criterion is not met when the area of ​​the predetermined item cannot be recognized from eight or more patterns, and determines that the recognition rate improvement criterion for the predetermined item is met based on the determination that the predetermined criterion is not met.

[0073] In addition, in this embodiment, as shown in FIG. 1, a case has been described in which the photographing device 53 photographs the work area 51 of the worker 61 located upstream or downstream of the conveyor 4, and the photographing device 54 photographs the work area 52 of the picking robot 62 located downstream or upstream of the conveyor 4, but a further supplement will be provided.

[0074] For example, with work areas 51 and 52 arranged on either side of a conveyor 4, a camera 53 may capture an image of the work area 51 of a worker 61, and a camera 54 may capture an image of the work area 52 of a picking robot 62.

[0075] Alternatively, one imaging device may image the work of the worker 61 in a work area and the work of the picking robot 62 in the same work area. In this case, the worker 61 and the picking robot 62 may coexist or may be interchangeable.

[0076] In the case of coexistence, the control device 3 determines whether the worker 61 is working or not based on the photographed image data output from the photographing device, and outputs a control signal to restrict or prohibit the operation of the picking robot 62 while the worker 61 is working. This makes it possible to prevent accidents during coexistence. In the case of replacement, the control device 3 detects whether the picking robot 62 is moving or waiting based on the photographed image data output from the photographing device, and outputs a control signal to restrict or prohibit the operation of the picking robot 62.

[0077] In addition, cameras of the same manufacturer and model are used for the photographing devices 53 and 54. Furthermore, the photographing environments of the work areas 51 and 52 are made the same. The same photographing environments are created by placing the same lighting equipment at the same distance from the subject to be photographed. As a result, the images of the same items included in the photographed image data output from the photographing devices 53 and 54 are the same. This makes it possible to improve the recognition accuracy.

[0078] According to the present embodiment, it is possible to provide a data processing device, a data processing method, and a data processing program that improve the efficiency of re-learning or additional learning for generating dictionary data used in recognition processing.

[0079] The data processing device 2 determines whether or not registering a specific registration candidate image data satisfies the recognition rate improvement standard for a specific item, and if the recognition rate improvement standard is met, registers the specific registration candidate image data as learning data, and if not, does not register the specific registration candidate image data as learning data. In this way, by registering registration candidate image data that satisfies the recognition rate improvement standard into the learning data, it is possible to improve the recognition rate. Also, by not registering registration candidate image data that does not satisfy the recognition rate improvement standard into the learning data, it is possible to reduce inefficient teaching processes.

[0080] The data processing device 2 refers to the auxiliary information and determines whether or not registering the predetermined registration candidate image data satisfies the recognition rate improvement standard for the predetermined item based on the dictionary data. If the item included in the dictionary data and the item included in the predetermined registration candidate image data do not match at all, even though they are supposed to be the same type of item, there is a possibility that the item area included in the predetermined registration candidate image data has not been correctly recognized. For example, if the outlines of both items do not match at all, there is a possibility that the outlines of multiple items included in the predetermined registration candidate image data have been erroneously detected as the outline of a single item. Such registration candidate image data is determined not to meet the recognition rate improvement standard and is not registered in the learning data. This makes it possible to reduce inefficient teaching processes.

[0081] Furthermore, the data processing device 2 determines whether or not registering the specified registration candidate image data satisfies the recognition rate improvement criteria for the specified item based on the recognition history of the specified item using the dictionary data. For an item that has failed to be recognized or has a low recognition success rate, there is a high possibility that the recognition rate will improve by additionally registering the registration candidate image data. Conversely, for an item that has been successfully recognized or has a high recognition success rate, there is a low possibility that the recognition rate will improve even if additional registration candidate image data is registered. In this way, by utilizing the recognition history, it is possible to more effectively register the registration candidate image data in the learning data.

[0082] Furthermore, the processor 21 of the data processing device 2 may detect a recognition rate improvement prediction score when determining whether or not the registration of a predetermined registration candidate image data satisfies the recognition rate improvement standard for a predetermined item. For example, the processor 21 detects a higher score as the number of recognition failures increases. The processor 21 associates the recognition rate improvement prediction score with the predetermined registration candidate image data to be registered in the learning data and registers it. When generating dictionary data based on the learning data, the efficiency of the teaching process can be improved by preferentially treating data with a high recognition rate improvement prediction score from the learning data as a teaching target.

[0083] The program according to this embodiment may be transferred in a state stored in an electronic device such as the data processing device 2, or in a state not stored in the electronic device. In the latter case, the program may be transferred via a network, or in a state stored in a storage medium. The storage medium is a non-transient tangible medium. The storage medium is a computer-readable medium. The storage medium may be an optical disk, a memory card, or any other medium capable of storing a program and being readable by a computer, and the form of the storage medium is not important. The electronic device downloads the program transferred (provided) via a network and installs it in memory, or reads the program from the storage medium and installs it in memory.

[0084] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0085] 1. Control system 2...Data processing device 3. Control device 4...Conveyor 7. Container 8. Items 9. Leader 21…Processor 22…Memory 23...Interface 31…Processor 32…Memory 33...Interface 51, 52…Work area 53, 54…Photographing device 61...Worker 62...Picking robot 211...input section 213...Judgment section 214…Registration Department 215...Recognition section 221…Administrative Information Area 222…Learning data area 223...Dictionary data area 311... Input section 312...Recognition section 313...Control section 321…Administrative information area 322...Dictionary data area

Claims

1. An interface for inputting predetermined registration candidate image data relating to a predetermined item; a processor that determines whether or not registering the predetermined registration candidate image data satisfies a criterion for improving the recognition rate of the predetermined item; A data processing device comprising:

2. The data processing device of claim 1 , wherein the processor determines whether the recognition rate improvement standard is satisfied based on the predetermined registration candidate image data and dictionary data for recognizing the predetermined item by image processing.

3. The data processing device of claim 2 , wherein the processor determines whether or not the recognition rate improvement standard is satisfied based on a recognition result using the predetermined registration candidate image data and the dictionary data.

4. The dictionary data is generated from learning data and includes parameters for recognizing the predetermined items; The data processing device according to claim 2 , wherein the processor registers the predetermined registration candidate image data as data to be used for learning when the recognition rate improvement standard is satisfied.

5. 5. The data processing device of claim 4, wherein said processor generates said parameters based on said predetermined registration candidate image data and said dictionary data, and updates said dictionary data when said recognition rate improvement standard is satisfied.

6. 4. The data processing device according to claim 2, wherein the processor determines whether or not registering the specified registration candidate image data satisfies the recognition rate improvement standard for the specified item based on a recognition history of the specified item using the dictionary data.

7. 4. The data processing device according to claim 2, wherein the processor stores a plurality of the predetermined registration candidate image data for the predetermined item, determines whether the judgment result for the plurality of predetermined registration candidate image data satisfies a predetermined criterion based on the plurality of predetermined registration candidate image data and the dictionary data, and determines whether the recognition rate improvement criterion for the predetermined item is satisfied based on whether the predetermined criterion is satisfied.

8. 4. The data processing device according to claim 3, wherein the processor determines that the recognition rate improvement standard is met when the consistency between the recognition result using the specified registration candidate image data and the dictionary data and the recognition result using data processed from the specified registration candidate image data and the dictionary data satisfies a consistency standard.

9. 9. The data processing device of claim 8, wherein the processor determines that the recognition rate improvement standard is met when the consistency between the contour of a specified item recognized from the specified registration candidate image data and the dictionary data and the contour of a specified item recognized from data processed from the specified registration candidate image data and the dictionary data satisfies a consistency standard.

10. the interface inputs predetermined auxiliary information related to a predetermined item together with the predetermined registration candidate image data; 4. The data processing device according to claim 2, wherein the processor determines, based on the predetermined auxiliary information, whether or not registering the predetermined registration candidate image data of an item of the same type as an item that is the target of recognition by the dictionary data satisfies the recognition rate improvement standard.

11. The auxiliary information includes size information of the predetermined item, The data processing device of claim 10 , wherein the processor determines, based on the size information, whether or not registering the predetermined registration candidate image data of an item of the same type as an item to be recognized by the dictionary data satisfies the recognition rate improvement standard.

12. The auxiliary information includes appearance information of the predetermined item, The data processing device of claim 10 , wherein the processor determines, based on the appearance information, whether or not registering the predetermined registration candidate image data of an item of the same type as an item that is a target of recognition in the dictionary data satisfies the recognition rate improvement standard.

13. the predetermined registration candidate image data is first photographed image data obtained by photographing a first area, The data processing device of claim 2 , wherein the processor recognizes the item from second photographed image data obtained by photographing a second area and the dictionary data.

14. the first area is an area where the predetermined item is handled by a worker, the second area is an area where the predetermined item is processed by a robot; The data processing device according to claim 13 , wherein the robot processes the predetermined item recognized from the second captured image data based on the dictionary data.

15. inputting predetermined registration candidate image data relating to a predetermined item; determining whether or not registering the predetermined registration candidate image data satisfies a criterion for improving a recognition rate of the predetermined item; A data processing method comprising:

16. On the computer, inputting predetermined registration candidate image data relating to a predetermined item; determining whether or not registering the predetermined registration candidate image data satisfies a criterion for improving a recognition rate of the predetermined item; A data processing program for executing the above.

Images