Processing apparatus and processing system

Abstract

To provide a processing apparatus in which usability for a user is ensured in classification of articles.SOLUTION: A processing apparatus comprises an image processing unit and a classification processing unit. The image processing unit divides a target area of a capture photographed image into a plurality of division areas. The classification processing unit selects only a part of the plurality of division areas and performs classification for the selected division area, and stores a classification result for the selected division area.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] FIELD OF THE INVENTION Embodiments of the present invention relate to processing devices and processing systems. [Background technology]

[0002] Processing devices are used to classify items such as agricultural produce, food, and manufactured products. In classifying items using a processing device, for example, the item to be classified is photographed as a subject, and the photographed image of the item is used to perform classification processing. In this process, a target area is extracted from the photographed image of the item, and the extracted target area is classified based on color information, etc. In classifying items in the manner described above, it is necessary to ensure usability for users who use the processing device to perform classification. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4171806 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a processing device and a processing system that ensures usability for users and the like in sorting items. [Means for solving the problem]

[0005] According to an embodiment, the processing device includes an image processing unit and a classification processing unit. The image processing unit divides a target area of ​​a captured image into a plurality of divided areas. The classification processing unit selects only a portion of the plurality of divided areas, performs classification for the selected divided areas, and saves the classification results for the selected divided areas. [Effects of the Invention]

[0006] According to the present invention, it is possible to provide a processing device and a processing system that ensures usability for users in sorting items. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram schematically illustrating an example of a processing system according to an embodiment. [Figure 2] FIG. 2 is a flowchart schematically illustrating an example of image processing performed on a captured image (original image) by the image processing unit of the processing execution unit in the embodiment. [Figure 3] FIG. 3 is a schematic diagram illustrating an example of a change caused by image processing of a captured image in the image processing according to the example of FIG. [Figure 4] FIG. 4 is a flowchart showing an example of classification processing for a target region of a captured image (original image), which is performed by the classification processing unit of the processing execution unit in the embodiment. [Figure 5] FIG. 5 is a schematic diagram illustrating an example of an order in which some divided regions are selected from a plurality of divided regions in the classification process according to the example of FIG. [Figure 6] FIG. 6 is a schematic diagram showing an example of divided areas selected in each of a plurality of frames captured consecutively when the divided areas are selected in the example order of FIG. [Figure 7] FIG. 7 is a schematic diagram illustrating an example of a classification result for each of a plurality of stored divided regions in the classification process according to the example of FIG. [Figure 8] Figure 8 is a schematic diagram illustrating an example of deriving the classification result for the entire target area and the likelihood of the classification result for the target area when the classification result shown in the example of Figure 7 is saved for each of multiple divided areas. [Figure 9] FIG. 9 is a schematic diagram illustrating an example of notification of notification information regarding classification when the classification result for the entire target region and the likelihood of the classification result for the target region are derived as in the example of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0008] The processing device (10) of the embodiment includes an image processing unit (21) and a classification processing unit (22). The image processing unit (21) divides a target area (T1) of a captured image (I1) into a plurality of divided areas (D1 to D9). The classification processing unit (22) selects only a portion of the plurality of divided areas (D1 to D9), performs classification for the selected divided areas, and saves the classification results for the selected divided areas. This effectively prevents delays in the classification process in the processing execution unit (11) in response to frame switching during capture by the imaging unit (15), ensuring usability for users who use the processing device (10) to perform classification.

[0009] In the processing device (10) of this embodiment, an image processing unit (21) divides a target region (T1) into multiple divided regions (D1 to D9) in each of multiple frames captured consecutively in capturing an image (I1). A classification processing unit (22) sequentially changes the selected divided region for each of the multiple frames, classifies the selected divided region for each of the multiple frames, and stores the classification results. As a result, classification of the target region (T1) as a whole is performed based on the classification results for each of the multiple divided regions (D1 to D9), and therefore classification of the target region (T1) is performed appropriately. Furthermore, since classification results are derived for all of the multiple divided regions (D1 to D9), the likelihood of the classification result for the target region (T1) can also be derived.

[0010] In the processing device (10) of the embodiment, the classification processing unit (22) generates notification information regarding classification indicating the classification result for the entire target region (T1) based on the stored classification results for each of the multiple divided regions (D1 to D9), in response to the classification results being stored for all of the multiple divided regions (D1 to D9), and notifies the generated notification information. Therefore, the classification result for the target region (T1) is appropriately derived based on the classification results for all of the multiple divided regions (D1 to D9). The appropriately derived classification result is then notified to the user of the processing device (10), etc.

[0011] In the processing device (10) of the embodiment, the classification processing unit (22) updates the stored classification results to the classification results of the classification performed in real time in response to the classification performed for one or more divided areas for which the classification results are stored. As a result, the latest classification results are appropriately stored for each of the multiple divided areas (D1 to D9), and appropriate classification results corresponding to the latest classification results for each of the divided areas (D1 to D9) are derived as the classification results for the entire target area (T1).

[0012] The processing system (1) of the embodiment includes an imaging unit (15), an image processing unit (21), and a classification processing unit (22). The image processing unit (21) divides a target area (T1) of an image (I1) captured by the imaging unit (15) into a plurality of divided areas (D1 to D9). The classification processing unit (22) selects only a portion of the plurality of divided areas (D1 to D9), performs classification for the selected divided areas, and saves the classification results for the selected divided areas. This effectively prevents delays in the classification process in the processing execution unit (11) in response to frame switching during imaging by the imaging unit (15), ensuring usability for users who use the processing system (1) to perform classification.

[0013] Hereinafter, embodiments will be described with reference to the drawings.

[0014] FIG. 1 is a block diagram illustrating an example of a processing system 1 according to an embodiment. As illustrated in FIG. 1, the processing system 1 includes a processing device (classification device) 10. The processing device 10 includes a processing execution unit 11, a data storage unit 12, a communication interface 13, an image capture unit 15, and a notification unit 16. The processing execution unit 11 includes an image processing unit 21 and a classification processing unit 22. In the example illustrated in FIG. 1, the image processing unit 21 includes a region extraction unit 23 and a region division unit 25, and the classification processing unit 22 includes a region selection unit 26, a determination unit 27, and a notification control unit 28. The image processing unit 21 and the classification processing unit 22 each perform a part of the processing performed by the processing execution unit 11.

[0015] In one example, the processing device 10 is, for example, a mobile terminal such as a smartphone or a computer, and includes a processor or an integrated circuit and a storage medium such as a memory. In the processing device 10, the processor or the integrated circuit functions as a processing execution unit 11, and the storage medium functions as a data storage unit 12. The processor or the integrated circuit of the processing device 10 includes any of a central processing unit (CPU), an application specific integrated circuit (ASIC), a microcomputer, a field programmable gate array (FPGA), a digital signal processor (DSP), and the like. The processing device 10 may include only one integrated circuit or the like, or may include multiple integrated circuits or the like. Furthermore, the processing device 10 may include only one storage medium or may include multiple storage media.

[0016] In the processing device 10, a processor or the like executes a program or the like stored in a storage medium or the like, thereby performing the processing described below by the processing execution unit 11. In one example, the program executed by the processor or the like in the processing device 10 may be stored in a computer (server) connected via a network such as the Internet, or in a server or the like in a cloud environment. In this case, the processor or the like of the processing device 10 downloads the program via the network. For example, if the processing device 10 is a terminal device such as a smartphone, downloading a dedicated application via the network enables the processor or the like of the processing device 10 to perform the processing described below by the processing execution unit 11.

[0017] In one example, at least a part of the processing system 1 (processing device 10) is configured from a server in a cloud environment. The infrastructure of the cloud environment is configured from a virtual processor such as a virtual CPU and a cloud memory. In the server in the cloud environment that serves as the processing device 10, the virtual processor or the like functions as a processing execution unit 11, and the virtual processor or the like performs the processing described below by the processing execution unit 11. The cloud memory functions as a data storage unit 12.

[0018] In the processing system 1, the processing device 10 may communicate with other processing devices via the communication interface 13. In this case, the processing device 10 can receive information from the other processing devices and can also transmit information to the other processing devices. The photographing unit 15 is composed of a camera, a video camera, or the like. In the processing device 10, the photographing unit 15 photographs a subject. The photographing unit 15 may be provided separately from the processing device 10.

[0019] In the processing device 10, the notification unit 16 constitutes a part of the user interface. The notification unit 16 can notify the user of the processing system 1 and the processing device 10 of information, for example, by displaying the information on a screen. In one example, the notification unit 16 is constituted by a touch panel or the like. Information can be displayed on the notification unit 16, and the user of the processing device 10 can input operations on the notification unit 16. In this case, the notification unit 16 is constituted by, for example, a display screen of a mobile terminal.

[0020] In addition, in the processing device 10, operation members such as buttons and switches may be provided separately from the notification unit 16, and operations may be input by the user of the processing device 10 or the like via the operation members. In this case, the operation members form part of the user interface. In one example, a user interface including the notification unit 16 may be provided separately from the processing device 10. In this case, the notification unit 16 is formed of a monitor or the like separate from the processing device 10. In another example, the notification unit 16 may notify information by emitting a sound instead of or in addition to a screen display.

[0021] In the processing system 1 of the embodiment, the processing device 10 classifies items such as agricultural products, food products, and manufactured products. In one example, the processing device 10 classifies items based on color information or the like. Items classified using the processing device 10 include vegetables and fruits, as well as meat, processed foods, alcoholic beverages, and dairy products. Vegetables classified using the processing device 10 include tomatoes, cabbage, and bell peppers, while fruits classified using the processing device 10 include citrus fruits such as mandarins, apples, and grapes. Items classified using the processing device 10 also include non-food products such as painted products and baked goods. Here, classification refers to sorting items into two or more categories based on predetermined criteria. Examples of predetermined criteria for classification include the variety of the item to be classified, the size or weight of the item to be classified, the quality or grade of the item to be classified, and the freshness of the item to be classified.

[0022] In the embodiment, the photographing unit 15 photographs the item to be classified, and the classification process is performed using the photographed image in which the item to be classified is the subject. In the following description, the image photographed by the photographing unit 15 is also referred to as the "original image." In one example, the category to which the item to be classified belongs is divided into a plurality of classes. Then, the process execution unit 11 determines to which of the plurality of divided classes the item to be classified belongs.

[0023] Here, a "category" and a "class" are each defined as a division of a category, and a "category" is a larger division than a "class." For example, "items" are divided into multiple categories, and one category is divided into multiple classes. Furthermore, the term "category" can be replaced with a "major division," and the term "class" can be replaced with a "minor division."

[0024] The category to which the item to be classified belongs is set, for example, based on an operation on a user interface by a user of the processing device 10. For example, a type of fruit such as mandarin oranges and apples, or a type of vegetable such as tomatoes and cabbage, may be set as the category to which the item to be classified belongs. In another example, a type of meat food such as beef and chicken, a type of painted product such as painted walls, or a type of pottery such as ceramics may be set as the category to which the item to be classified belongs.

[0025] In classifying items, an image is taken of the item to be classified as a subject. When taking the image, a plurality of frames are taken continuously over time. That is, the images are taken with the frames switched at a predetermined frame rate. Furthermore, the same item is taken as a subject in each of the consecutively taken frames. The image processing unit 21 of the processing execution unit 11 acquires the consecutively taken frames by the image capture unit 15 and performs image processing on each of the acquired frames.

[0026] 2 is a flowchart showing an example of image processing performed on a captured image (original image) by the image processing unit 21 of the processing execution unit 11 in the embodiment. In the embodiment, each time a frame is captured, the image processing shown in the example of FIG. 2 is performed on the captured frame. That is, the image processing shown in the example of FIG. 2 is performed on each of a plurality of frames captured consecutively over time.

[0027] When the processing of the example of FIG. 2 starts, the area extraction unit 23 of the image processing unit 21 extracts a target area in the captured image as an area to be subjected to classification processing (S101). At this time, for example, a partial area of ​​an item shown in the captured image is extracted as the target area. By extracting the target area, an extracted image in which the target area is extracted from the captured image (original image) is generated. In one example, a surrounding frame is set on the capture screen at the time of capture. Then, the area surrounded by the surrounding frame in the captured image is extracted as the target area. In another example, the target area to be subjected to classification processing is identified in the captured image by performing image analysis of the captured image using a machine learning model.

[0028] In classifying the items to be classified, the images are taken and the target areas are extracted so that there is no or almost no deviation in the areas extracted as target areas in the items between multiple frames taken in succession. In other words, the images are taken and the target areas are extracted so that the same or nearly the same areas are extracted as target areas from the items to be classified between multiple frames taken in succession.

[0029] Then, the region dividing unit 25 of the image processing unit 21 divides the extracted target region into a plurality of divided regions (S102). By dividing the target region into a plurality of divided regions, the extracted image showing the extracted target region is divided into a plurality of divided images, and a divided image group, which is a collection of the divided images, is generated. Note that the number of divided regions divided from the target region is not particularly limited as long as it is two or more. In one example, the target region is divided into three in the vertical direction of the captured image and three in the horizontal direction of the captured image. As a result, the target region is divided into 3 x 3, or nine divided regions. In another example, the target region may be divided into 1 x 3 or 5 x 3.

[0030] Fig. 3 is a schematic diagram illustrating an example of changes caused by image processing of a captured image in image processing according to the example of Fig. 2. In the example of Fig. 3, an apple 3 is captured as an item to be classified, and a captured image I1 showing the apple 3 is captured. Then, a target region T1 is extracted from the captured image I1 of the apple 3. At this time, a partial region of the apple 3 is extracted as the target region T1. Then, the extracted target region T1 is divided into 3 x 3 regions, which are divided into nine divided regions D1 to D9.

[0031] Fig. 4 is a flowchart showing an example of classification processing for a target area of ​​a captured image (original image) performed by the classification processing unit 22 of the processing execution unit 11 in the embodiment. In the embodiment, each time a frame is captured, the image processing shown in Fig. 2 is performed on the captured frame, and then the classification processing shown in Fig. 4 is performed. That is, the image processing shown in Fig. 2 is performed on each of a plurality of frames captured continuously over time, and then the classification processing shown in Fig. 4 is performed. Therefore, the classification processing shown in Fig. 4 is performed on a target area divided into a plurality of divided areas.

[0032] When the processing of the example of FIG. 4 starts, the area selection unit 26 of the classification processing unit 22 selects only some of the divided areas of the target area as areas to be classified (S111). In one example, one divided area of ​​the multiple divided areas is selected. In another example, the target area is divided into three or more divided areas, and only some of the three or more divided areas are selected. In this case, two or more divided areas may be selected. However, in the processing of S111, not all of the multiple divided areas divided from the target area are selected.

[0033] Furthermore, in the process of S111, the region selection unit 26 sequentially changes the selected divided region for each frame in a plurality of frames captured consecutively. In one example, the region selection unit 26 selects a divided region different from the divided region selected in the immediately preceding frame. In another example, the region selection unit 26 selects the same divided region in two or more consecutive frames, and then selects a divided region in the next frame that is different from the divided region selected in the immediately preceding frame.

[0034] The order in which some divided areas are selected from the plurality of divided areas is not particularly limited. In one example, the selected divided areas are changed for each frame in a predetermined order. In another example, the selected divided areas are changed randomly for each frame. Furthermore, the number of frames selected by the process of S111 from the first frame to the last frame captured may be equal or unequal for the plurality of divided areas. However, in either case, from the first frame to the last frame captured, each of all divided areas is selected as an area to be classified in one or more frames.

[0035] Then, the determination unit 27 of the classification processing unit 22 performs classification by making a classification determination for the selected divided area (S112). The classification of the selected divided area is performed, for example, based on the color information of the divided area, and a classification determination for the divided area is performed using an appropriate method used for classification. In one example, an algorithm for classifying the category to which the target product belongs into multiple classes using any method such as a decision tree or an SVM (support vector machine) is stored in the data storage unit 12. Then, the determination unit 27 classifies the selected divided area using the algorithm. In another example, the determination unit 27 uses a machine learning model to perform image analysis of the image showing the selected divided area, and classifies the selected divided area.

[0036] As described above, the selected divided area is changed sequentially for each frame. Therefore, the divided area for which the classification determination is performed is switched sequentially for each frame. Furthermore, the classification determination in S112 is performed only for the divided area selected in the process of S111, and is not performed for the divided area not selected in the process of S111.

[0037] Then, the determination unit 27 of the classification processing unit 22 saves the classification result for the selected divided area (S113). The classification result may be saved in a storage medium that becomes the data storage unit 12, or may be stored in a storage area of ​​a processor or the like that constitutes the processing execution unit 11. The classification result for the selected divided area indicates, for example, the class to which the selected divided area belongs among a plurality of classes into which the categories are divided. For divided areas for which classification results have not been saved, the classification result from the classification performed in real time is saved as the initial classification result.

[0038] Furthermore, for a segmented area for which classification results have already been saved, the saved classification results are updated to the classification results of the classification performed in real time. At this time, if the classification results of the classification performed in real time differ from the classification results that have already been saved, the saved classification results are changed by the update. On the other hand, if the classification results of the classification performed in real time are the same as the classification results that have already been saved, the saved classification results are not changed even if the update is performed. Furthermore, in one example, if the classification results of the classification performed in real time are the same as the classification results that have already been saved, the saved classification results may be maintained without being updated to the classification results of the classification performed in real time.

[0039] Then, the determination unit 27 of the classification processing unit 22 determines whether or not classification results have been saved for all of the multiple divided areas (S114). If there is a divided area for which classification results have not been saved (S114-No), the process of the example of FIG. 4 ends. On the other hand, if classification results have been saved for all of the divided areas (S114-Yes), the determination unit 27 classifies the entire target area by, for example, making a classification determination (S115). At this time, the determination unit 27 classifies the target area based on the classification results for each of the multiple divided areas that have been saved. The classification result for the entire target area indicates information similar to that for the classification result for the divided areas, for example, the class to which the target area belongs out of the multiple classes into which the categories are divided.

[0040] In the example process of FIG. 4 , once the classification result for the target region is derived, the determination unit 27 of the classification processing unit 22 derives the likelihood (degree of appropriateness) of the classification result for the target region (S116). At this time, the determination unit 27 derives the likelihood of the classification result for the target region based on the classification results for each of the saved divided regions. Then, the notification control unit 28 of the classification processing unit 22 generates notification information related to the classification indicating the classification result for the target region and the likelihood of the classification result for the target region (S117). Then, the notification control unit 28 of the classification processing unit 22 notifies the notification unit 16 of the generated notification information (S118). At this time, the notification unit 16 notifies the notification information, for example, by displaying the notification information on a screen.

[0041] In the embodiment, by performing the classification process of the example of Fig. 4, the classification processing unit 22 selects only some of the multiple divided areas, performs classification for the selected divided areas, and saves the classification results for the selected divided areas. Then, by performing the classification process of the example of Fig. 4 for each of multiple frames captured continuously, the classification processing unit 22 sequentially changes the selected divided areas for each of the multiple frames, classifies the selected divided areas for each of the multiple frames, and saves the classification results.

[0042] 4, the classification processing unit 22 generates notification information related to classification indicating the classification result for the entire target area based on the classification results for each of the plurality of divided areas stored, corresponding to the classification results being stored for all of the plurality of divided areas, and notifies the generated notification information. Furthermore, by performing the classification processing of the example of FIG. 4, the classification processing unit 22 updates the stored classification result to the classification result obtained in real time, corresponding to the classification being performed for one or more divided areas for which the classification results are stored.

[0043] FIG. 5 is a schematic diagram illustrating an example of the order in which some divided regions are selected from a plurality of divided regions in the classification process according to the example of FIG. 4. In the example of FIG. 5, similar to the example of FIG. 3, the target region T1 is divided into nine divided regions D1 to D9. Then, in one frame, one of the nine divided regions D1 to D9 is selected as the region to be classified. Also, in the example of FIG. 5, the selected divided region is sequentially changed for each frame so that the number of selected frames is uniform among the divided regions D1 to D9. Then, the selected divided region is changed in the order of divided regions D1, D2, D3, ..., D8, D9 (arrow C1). Then, in the frame following the frame in which divided region D9 is selected, divided region D1 is selected again (arrow C2).

[0044] Fig. 6 is a schematic diagram showing an example of the divided areas selected in each of a plurality of frames captured consecutively when the divided areas are selected in the order shown in Fig. 5. In the example shown in Fig. 6, when photographing an item to be sorted, a plurality of frames are captured consecutively in the order of frames F1, F2, F3, ... starting with the first frame F1. In Fig. 6, the divided areas selected in each of the frames are shown in black.

[0045] In the example of Fig. 6, divided regions D1, D2, ..., D9 are selected in frames F1, F2, ..., F9, respectively. Then, divided regions D1, D2, ..., D9 are selected in frames F10, F11, ..., F18, respectively. In the example of Fig. 6, classification is performed for divided region D9 in frame F9, and classification results are saved for all nine divided regions D1 to D9. Therefore, classification is performed for divided region D9 in frame F9, and notification information related to classification is generated, which indicates the classification result for the entire target region T1, based on the saved classification results for each of divided regions D1 to D9.

[0046] 6, in each of the frames after frame F10, classification is performed on the divided regions for which classification results have already been saved. Therefore, in each of the frames after frame F10, the saved classification results for the divided regions for which real-time classification has been performed are updated to the classification results of the classification performed in real time. In one example, similar to the example of FIG. 6, divided regions are selected in each of multiple frames. Then, in each of the frames after frame F10, the classification results for the target region T1 and the likelihood of the classification results for the target region T1 are updated in response to the updated classification results for the divided regions for which real-time classification has been performed. Then, notification information indicating the updated classification results and likelihoods is generated.

[0047] FIG. 7 is a schematic diagram illustrating an example of classification results for each of a plurality of divided areas stored in the classification process according to the example of FIG. 4. In the example of FIG. 7, one of nine divided areas D1 to D9 is selected in each of a plurality of frames captured consecutively in the same order as the examples of FIGS. 5 and 6. Also, in the example of FIG. 7, classification results are stored for all of the nine divided areas D1 to D9. In the example of FIG. 7, the category to which the item to be classified belongs is divided into three classes A1 to A3, and one of the classes A1 to A3 is displayed as the classification result. In the example of FIG. 7, the stored classification results for divided areas D1, D2, D3, ..., D8, D9 are classes A1, A2, A1, A2, A2, A1, A1, A1, and A1, respectively.

[0048] FIG. 8 is a schematic diagram illustrating an example of deriving a classification result for the entire target region and the likelihood of the classification result for the target region when the classification results shown in the example of FIG. 7 are saved for each of the multiple divided regions. In the example of FIG. 8, in deriving the classification result for the target region, the number of divided regions classified into each of classes A1 to A3 is derived from divided regions D1 to D9. In the example of FIG. 8, of the nine divided regions D1 to D9, six are classified into class A1 and three are classified into class A2. Then, the one with the largest cumulative number of divided regions classified into classes A1 to A3 is derived as the classification result for target region T1. Therefore, in the example of FIG. 8, class A1 is derived as the classification result for target region T1.

[0049] Furthermore, in deriving the likelihood of the classification result of the target region, for each of classes A1 to A3, the ratio of the number of classified divided regions to the number of divided regions divided from the target region is derived as the probability that the target region belongs to that class. Then, of the three probabilities (ratios) for classes A1 to A3, the probability derived for the class that will be the classification result of the target region is derived as the likelihood of the classification result of the target region. In the example of FIG. 8, for classes A1, A2, and A3, the ratios of the number of classified divided regions to the number of divided regions divided from the target region, i.e., the probabilities that the target region belongs to that class, are 2 / 3, 1 / 3, and 0, respectively. Then, 2 / 3, the probability for class A1, is derived as the likelihood of the classification result of the target region.

[0050] FIG. 9 is a schematic diagram illustrating an example of notification of notification information related to classification when the classification result for the entire target region and the likelihood of the classification result for the target region are derived as in the example of FIG. 8. In the example of FIG. 9, the classification result for the target region and the likelihood of the classification result are notified by a screen display. The classification result for the target region is displayed by display element 5, and the likelihood of the classification result, etc. are displayed by display element 6. In the example of FIG. 9, the classification result for the target region is displayed in display element 5, indicating that the target region has been classified into class A1. In addition, in the example of FIG. 9, the proportion of the area occupied by class A1 to the area inside frame 7 of display element 6 changes in accordance with the likelihood of the classification result.

[0051] 9, when the classification result is class A1, the ratio of the area occupied by class A2 to the area inside the enclosing frame 7 of the display element 6 changes in response to the probability that the target region belongs to class A2. And the ratio of the area occupied by class A3 to the area inside the enclosing frame 7 of the display element 6 changes in response to the probability that the target region belongs to class A3.

[0052] In one example, the notification information may indicate the probability that the target region belongs to each of multiple classes, including the likelihood of the classification result, using a numerical value or the like. In another example, only the classification result for the target region may be derived, and the likelihood of the classification result may not be derived. In this case, the notification information does not indicate the likelihood or the like.

[0053] As described above, in the embodiments, the classification processing unit 22 selects only some of the multiple divided regions obtained by dividing the target region, and performs classification and saves the classification results for the selected divided regions. Therefore, in the embodiments, the processing amount per frame in the classification process is reduced compared to when classification is performed for all divided regions. This reduces the load on the processor, etc., constituting the processing execution unit 11 in the processing device 10, and effectively prevents delays in the classification process in the processing execution unit 11 relative to frame changes during image capture by the image capture unit 15. Since the classification process is not delayed relative to frame changes during image capture, notification information including the classification results for the target region is appropriately notified to the user of the processing device 10, etc., in accordance with frame changes during image capture. For example, the notification information is notified to the user of the processing device 10, etc., without causing any stuttering in the display image of the notification information. Therefore, usability for users, etc., who perform classification using the processing device 10 is ensured.

[0054] In some embodiments, the classification processing unit 22 sequentially changes the selected divided area for each of the multiple frames captured consecutively, classifies the selected divided area for each of the multiple frames, and saves the classification results. This allows classification for all of the multiple divided areas while reducing the processing load per frame in the classification process. This allows classification of the entire target area based on the classification results for each of the multiple divided areas, thereby ensuring appropriate classification of the target area. Furthermore, since classification results are derived for all of the multiple divided areas, the likelihood of the classification results for the target area can also be derived, as described above. This allows users of the processing device 10 to understand not only the classification results for the target area, but also the likelihood (degree of appropriateness) of the classification results for the target area.

[0055] In one example of the embodiment, the classification processing unit 22 generates classification notification information indicating the classification result for the entire target area based on the classification results for each of the stored divided areas, in response to the classification results for all of the divided areas being stored. Therefore, the classification result for the target area is appropriately derived based on the classification results for all of the divided areas. The appropriately derived classification result is then notified to the user of the processing device 10.

[0056] In one example of the embodiment, the classification processing unit 22 updates the stored classification results to the classification results of the classification performed in real time in response to the classification performed for one or more divided areas for which the classification results are stored. As a result, the latest classification results are appropriately stored for each of the multiple divided areas, and appropriate classification results corresponding to the latest classification results for each divided area are derived as classification results for the entire target area.

[0057] In one modified example, the classification processing unit 22 assigns weights to each of a plurality of divided areas divided from the target area. In one example, data indicating the weights to be assigned to each of the divided areas is stored in the data storage unit 12, and the classification processing unit 22 assigns weights to each of the divided areas based on the stored data. In another example, the classification processing unit 22 sets the weights to be assigned to each of the plurality of divided areas based on the positions of each of the divided areas in the captured frame.

[0058] In this modification, the classification processing unit 22 increases the number of frames selected by the process of S111 described above for a divided area having a larger weight. As a result, in the process of S111, some divided areas are selected from among the multiple divided areas so that the number of frames selected for a divided area having a larger weight increases.

[0059] In one example, similar to the examples in Figures 5 and 6, the target region T1 is divided into nine divided regions D1 to D9. However, in this example, the weighting is performed such that the weights of the divided regions D1, D3, D7, and D9 are each set to 1, the weights of the divided regions D2, D4, D6, and D8 are each set to 2, and the weight of the divided region D5 is set to 3. In this example, in selecting some divided regions by the process of S111, the number of frames in which each of the divided regions D2, D4, D6, and D8 is selected is doubled compared to the number of frames in which each of the divided regions D1, D3, D7, and D9 is selected. In addition, the number of frames in which the divided region D5 is selected is tripled compared to the number of frames in which each of the divided regions D1, D3, D7, and D9 is selected.

[0060] In one modified example, the classification processing unit 22 derives a classification result for the entire target area and the likelihood of the classification result for the target area based on the weights assigned to each of the multiple divided areas, in addition to the classification results for each of the saved divided areas. In this modified example, in the processes of S115 and S116, the classification processing unit 22 derives the classification result for the target area and the likelihood of the classification result for the target area by increasing the contribution of a divided area with a larger weight to the classification result for the target area and the likelihood of the classification result for the target area.

[0061] In one example, the classification results stored for nine divided regions D1 to D9 are similar to the example in FIG. 7. Therefore, the classification results stored for divided regions D1, D2, D3, ..., D8, D9 are classes A1, A2, A1, A2, A2, A1, A1, A1, A1, and A1, respectively. However, in this example, the weights are assigned such that the weights of divided regions D1, D3, D7, and D9 are each set to 1, the weights of divided regions D2, D4, D6, and D8 are each set to 2, and the weight of divided region D5 is set to 3. Then, in deriving the classification result of the target region and the likelihood of the classification result of the target region, the contribution of each of divided regions D2, D4, D6, and D8 is doubled relative to the contribution of each of divided regions D1, D3, D7, and D9. Furthermore, the contribution of divided region D5 is tripled relative to the contribution of each of divided regions D1, D3, D7, and D9.

[0062] In this example, in addition to the classification results for each divided area, the probability that the target area belongs to each of classes A1 to A3 is derived based on the contribution of each divided area. Therefore, in this example, the probabilities that the target area belongs to classes A1, A2, and A3 are 8 / 15, 7 / 15, and 0, respectively. In this example, as in the example of Figure 8, class A1 is derived as the classification result for the target area. However, in this example, the likelihood of the classification result for the target area is 8 / 15, which is different from the example of Figure 8.

[0063] 4 is performed on each of a plurality of frames, and once the classification results for all of the divided areas are saved, classification is performed on some of the divided areas for each frame, and the saved classification results for the divided areas that have been classified in real time are updated. Then, for each frame, the classification result for the target area and the likelihood of the classification result for the target area are updated, and notification information indicating the updated classification result and likelihood is generated.

[0064] However, in one modified example, the classification processing unit 22 updates the classification result for the target region and the likelihood of the classification result for the target region for each reference number N (N is an integer equal to or greater than 2) of frames, and generates notification information indicating the updated classification result, etc. In one example, the reference number N is set to a value equal to the number of divisions into which the target region is divided. In another example, the reference number N is set based on the number of divisions into which the target region is divided and the weights assigned to each of the divisions.

[0065] In one example of this modified example, the target region T1 is divided into nine divided regions D1 to D9, and the reference number N is set to 9. Then, divided regions are selected in each of a plurality of frames, similar to the example of FIG. 6. In this example, once the classification result for the target region is derived in frame F9, updating of the classification result for the target region and generation of notification information are not performed between frames F10 to F17. Then, in frame F18, the classification result for the target region and the likelihood of the classification result for the target region are updated based on the classification results for each of the stored divided regions D1 to D9. Then, in frame F18, notification information indicating the updated classification result and likelihood is generated.

[0066] In another example of this modification, target region T1 is divided into nine divided regions D1 to D9. Then, by weighting, the weights of divided regions D1, D3, D7, and D9 are each set to 1, the weights of divided regions D2, D4, D6, and D8 are each set to 2, and the weight of divided region D5 is set to 3. In this example, between frames F1 to F15 and between frames F16 to F30, each of divided regions D1, D3, D7, and D9 is selected in one frame, each of divided regions D2, D4, D6, and D8 is selected in two frames, and divided region D5 is selected in three frames.

[0067] Also, in this example, the reference number N is set to 15. Then, once the classification result for the target region is derived in frame F15, updating of the classification result for the target region and generation of notification information are not performed between frames F16 to F29. Then, in frame F30, the classification result for the target region and the likelihood of the classification result for the target region are updated based on the classification results for each of the saved divided regions D1 to D9. Then, in frame F30, notification information indicating the updated classification result and likelihood is generated.

[0068] Furthermore, in the above-described embodiment, classification of the target area and generation of notification information are not performed until classification results are saved for all of the divided areas. However, in one modified example, the classification processing unit 22 performs classification of the target area and generation of notification information even when classification results are saved for only some of the divided areas. In this modified example, for divided areas for which classification results are not saved, the classification information is set to an initial value such as 0. Then, when the classification results are saved, the classification information of the divided area is changed from the initial value to, for example, one of classes A1 to A3, which is the classification result.

[0069] In one example of this modification, when classification results are stored for only some of the divided regions, the classification processing unit 22 derives the classification results of the target region and the likelihood of the classification results using only the classification information of the divided regions in which the classification results are stored. In this case, the classification information of the divided regions whose classification information is the initial value is not used for classifying the target region, etc.

[0070] For example, suppose the classification information for divided regions D1, D2, D3, ..., D8, and D9 is class A1, class A2, class A1, class A2, class A2, initial value, initial value, initial value, initial value, respectively. In this case, in this example, the classification result of the target region and the likelihood of the classification result are derived using only the classification results of divided regions D1 to D5. The classification information of divided regions D6 to D9 is not used for classifying the target region. Class A2 is then derived as the classification result for the target region. Furthermore, 3 / 5 is calculated as the likelihood of the classification result of the target region.

[0071] In another example of this modification, even when classification results are stored for only some of the divided areas, the classification processing unit 22 uses the classification information for all of the divided areas to derive the classification results, etc. for the target area. In this case, the classification information for the divided areas whose classification information is set to the initial value is also used to classify the target area, etc.

[0072] For example, for divided regions D1, D2, D3, ..., D8, and D9, the classification information is assumed to be class A1, class A2, class A1, class A2, class A2, initial value, initial value, initial value, initial value, respectively. In this case, in this example, an initial value such as 0 is derived as the classification result for the target region. Also, for example, for divided regions D1, D2, D3, ..., D8, and D9, the classification information is assumed to be class A1, class A2, class A1, class A2, class A2, class A1, class A1, initial value, initial value, respectively. In this case, in this example, class A1 is derived as the classification result for the target region.

[0073] According to at least one of these embodiments, the image processing unit divides the target area of ​​the captured image into a plurality of divided areas. The classification processing unit selects only a portion of the divided areas, performs classification for the selected divided areas, and saves the classification results for the selected divided areas. This makes it possible to provide a processing device and a processing system that ensures usability for users and others when classifying items.

[0074] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0075] 1...processing system, 10...processing device, 11...processing execution unit, 12...data storage unit, 15...photographing unit, 16...notification unit, 21...image processing unit, 22...classification processing unit, I1...captured image, T1...target area, D1 to D9...divided areas.

Claims

1. an image processing unit that divides a target area of ​​a captured image into a plurality of divided areas; a classification processing unit that selects only some of the plurality of divided regions, performs classification on the selected divided regions, and saves the classification results for the selected divided regions; A processing device comprising:

2. the image processing unit divides the target area into the plurality of divided areas in each of a plurality of frames captured consecutively in capturing a captured image; the classification processing unit sequentially changes the selected divided area for each of the plurality of frames, and performs the classification for the selected divided area for each of the plurality of frames and stores the classification result. The processing device of claim 1.

3. 3. The processing device of claim 2, wherein the classification processing unit generates notification information regarding classification indicating the classification result for the entire target area based on the classification results for each of the plurality of divided areas, in response to the classification results being stored for all of the plurality of divided areas, and notifies the generated notification information.

4. 4. The processing device according to claim 2, wherein the classification processing unit updates the stored classification results to the classification results of the classification performed in real time in response to the classification being performed for one or more of the divided areas in which the classification results are stored.

5. With the photography department; an image processing unit that divides a target area of ​​an image captured by the imaging unit into a plurality of divided areas; a classification processing unit that selects only some of the plurality of divided regions, performs classification on the selected divided regions, and saves the classification results for the selected divided regions; A processing system comprising:

Images