Information processing device, program and system

Abstract

To provide an information processing device capable of efficiently improving accuracy of an image recognition model.SOLUTION: An information processing device includes: a communication unit that communicates with a server having an image recognition model that recognizes an object from an image, the image recognition model constructed by machine-learning using an image including the object as teacher data; a display unit; an input unit that receives an operation of a user; and a control unit that causes the display unit to display an image of a recognition result obtained by the image recognition model from a captured image obtained by capturing an image of a first object, causes the input unit to receive an operation for correcting the recognition result, and transmits correction information representing the corrected recognition result to the server in order to correct the image recognition model using the teacher data including the correction information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a program, and a system. [Background technology]

[0002] Conventionally, techniques have been proposed to assist in counting counting objects such as steel materials. A counting device that performs counting processing using a shape matching model and displays an image of the counting results is known (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6127188 Summary of the Invention [Problem to be solved by the invention]

[0004] In image recognition models for counting objects, there is room for efficiently improving the accuracy of shape matching models.

[0005] In view of the above circumstances, an object of the present disclosure is to provide an information processing device and the like that can efficiently improve the accuracy of an image recognition model. [Means for solving the problem]

[0006] An information processing device according to an embodiment of the present disclosure includes: a communication unit that communicates with a server having an image recognition model that recognizes the object from an image, the image being constructed by machine learning using an image including the object as training data; A display unit; an input unit that accepts user operations; a control unit that causes the display unit to display an image of a recognition result obtained by the image recognition model from an image obtained by imaging a first object, receives an operation to correct the recognition result by the input unit, and transmits correction information indicating the corrected recognition result to the server to correct the image recognition model using the training data including the correction information; It has. [Effects of the Invention]

[0007] According to an embodiment of the present disclosure, it is possible to efficiently improve the accuracy of an image recognition model. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic diagram of an information processing system according to an embodiment of the present invention. [Figure 2] 10 is a flowchart illustrating an example of the operation of the information processing device. [Figure 3A] FIG. 10 is a diagram illustrating an example of a captured image. [Figure 3B] FIG. 10 is a diagram illustrating an example of a recognition result image. [Figure 3C] FIG. 2 is a diagram illustrating an example of a user interface provided by an information processing device. [Figure 3D] FIG. 2 is a diagram illustrating an example of a user interface provided by an information processing device. [Figure 3E] FIG. 2 is a diagram illustrating an example of a user interface provided by an information processing device. [Figure 3F] FIG. 2 is a diagram illustrating an example of a user interface provided by an information processing device. [Figure 4A] FIG. 10 is a diagram showing a modified example of the recognition result image. [Figure 4B] FIG. 10 is a diagram showing a modified example of the recognition result image. [Figure 4C] FIG. 10 is a diagram showing a modified example of the recognition result image. DETAILED DESCRIPTION OF THE INVENTION

[0009] FIG. 1 is a schematic diagram of an information processing system 4 according to this embodiment. The information processing system 4 includes one or more information processing devices 1 and one or more server devices 2 that can communicate with each other via a network 3. The network 3 includes, for example, a mobile communication network, the Internet, or a fixed communication network. The information processing device 1 and the server device 2 may be directly connected via a signal line or the like. The processes executed by the information processing device 1 and the server device 2 according to this embodiment may be executed by multiple information processing devices 1 and server devices 2 that are distributed. The information processing device 1 may communicate with or be connected to multiple server devices 2. The server device 2 may communicate with or be connected to multiple information processing devices 1. The information processing device 1 is, for example, a terminal device such as a mobile device, a mobile phone, a smartphone, a wearable device, or a tablet. The server device 2 is, for example, composed of one or more server computers. The server device 2 can implement any artificial intelligence using machine learning techniques, including neural networks and deep learning.

[0010] The information processing system 4 assists a user in counting objects placed at a work site. The user is an inspection worker or the like at the work site. The work site is, for example, an indoor or outdoor factory, a construction site, or other base where production, construction, or other work is performed. The user needs to identify and count the objects, i.e., count them, for inventory management, legal compliance, or the like. The objects are, for example, items manufactured in a factory, parts or tools used to manufacture items in a factory, or materials or tools used in construction work at a construction site. Factory items or parts include bolts, nuts, washers, etc., and tools include wrenches, etc. Furthermore, construction site materials include iron pipes, rebar, steel frames, explosives, etc., and tools include clamps, etc. A user uses an information processing device 1 at hand to send captured images of the objects to a server device 2, and information processing for counting the objects is executed by the server device 2.

[0011] In this embodiment, the information processing device 1 includes at least a communication unit 12, a display unit 14, an input unit 15, and a control unit 11. The communication unit 12 communicates with a server device 2 having an image recognition model 24 that recognizes objects from images, constructed through machine learning using images including the objects as training data. The display unit 14 displays various images to the user. The input unit 15 accepts user operations on the displayed images. The control unit 11 acquires a captured image obtained by capturing an image of a first object (hereinafter referred to as a recognition target object) placed at the user's work base using the imaging unit 16. The control unit 11 displays an image of the recognition result obtained by the image recognition model 24 from the captured image. The control unit 11 accepts operations for correcting the recognition result. The control unit 11 transmits correction information indicating the corrected recognition result to the server device 2 to correct the image recognition model 24 using the correction information. In actual operation at the work base, if an incorrect recognition result is obtained using the image recognition model 24, and therefore an accurate counting result is not obtained, the user can correct the recognition result. This correction to the recognition result is used as an annotation for machine learning the image recognition model 24. The information processing device 1 receives an operation from the user to correct the recognition result and transmits the correction information to the server device 2, thereby causing the server device 2 to instantly correct the image recognition model 24. This allows the image recognition model 24 in the actual operation stage to be trained at any time using the correction information. Therefore, it becomes possible to improve the accuracy of the image recognition model 24 in approximately real time while it is in use.

[0012] 1, the information processing device 1 includes a control unit 11, a communication unit 12, a display unit 14, and an input unit 15, as well as a storage unit 13 and an imaging unit 16. The components of the information processing device 1 are connected to each other so that they can communicate with each other, for example, via dedicated lines. Each unit will be described in detail below.

[0013] The control unit 11 is, for example, one or more general-purpose processors including a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit). The control unit 11 may include one or more dedicated processors specialized for specific processing. Instead of including a processor, the control unit 11 may include one or more dedicated circuits. The dedicated circuits may be, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 11 may include an ECU (Electronic Control Unit). The control unit 11 transmits and receives any information via the communication unit 12. The control unit 11 processes images, etc. transmitted from the imaging unit 16 or the server device 2, generates images to be displayed on the display unit 14, etc.

[0014] The communication unit 12 includes one or more communication modules for connecting to the network 3. The communication unit 12 may include a module compatible with one or more mobile communication standards including LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation). The communication unit 12 may include a communication module compatible with one or more short-range communication standards or specifications including Bluetooth (registered trademark), AirDrop (registered trademark), IrDA, ZigBee (registered trademark), FeliCa (registered trademark), or RFID. The communication unit 12 can be connected to the server device 2 or another computer, etc., via the network 3.

[0015] The storage unit 13 includes, for example, a semiconductor memory, a magnetic memory, an optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a random access memory (RAM) or a read-only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read-only memory (EEPROM). The storage unit 13 may function, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 13 may store, for example, information resulting from analysis or processing by the control unit 11 or the server device 2. The storage unit 13 may store various information related to the operation or control of the information processing device 1. The storage unit 13 may store system programs, application programs, embedded software, etc. The storage unit 13 may be provided outside the information processing device 1 and accessed by the information processing device 1.

[0016] The display unit 14 is, for example, a display. The display is, for example, an LCD (liquid crystal display) or an organic EL (electro luminescence) display. The display unit 14 may be provided outside the information processing device 1 and accessed from the information processing device 1. As a connection method, any method such as USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), or Bluetooth (registered trademark) can be used. The display unit 14 displays, for example, information on the results of analysis or processing by the control unit 11 or the server device 2.

[0017] The input unit 15 is, for example, a touch sensor, physical keys, capacitance keys, a pointing device, or a microphone that is integrated with a display. The input unit 15 may be provided externally to the information processing device 1 and accessed from the information processing device 1. Any connection method, for example, USB, HDMI (registered trademark), or Bluetooth (registered trademark) may be used. The input unit 15 accepts an operation to input information used in the operation of the information processing device 1. When the input unit 15 includes a touch sensor, the input unit 15 detects contact with a user's finger, a stylus pen, or the like, and identifies the contact position. The input unit 15 may be integrated with a display to form a touch panel display.

[0018] The imaging unit 16 includes a camera for capturing visible light images. The imaging unit 16 can capture images of any object. For image recognition, the imaging unit 16 stores the captured images in the storage unit 13 or transmits them to the control unit 11. The images include still images and videos. The imaging unit 16 may be configured as a part of the information processing device 1, or may be provided outside the information processing device 1 and connected to the information processing device 1 via wire or wirelessly so as to be able to send and receive control signals, images, etc.

[0019] The server device 2 includes a control unit 21, a communication unit 22, and a storage unit 23. The communication unit 22 has a LAN interface for communicating with the information processing device 1. The interface may include an interface for mobile communication. The server device 2 is connected to a network 3 via the communication unit 22 and communicates information with the information processing device 1 via the network 3. The same explanation as for the control unit 11 and the storage unit 13 of the information processing device 1 applies to the control unit 21 and the storage unit 23 of the server device 2. The storage unit 23 may store system programs, application programs, embedded software, etc. The storage unit 23 may store information resulting from analysis or processing by the control unit 21 or the information processing device 1. The control unit 21 transmits and receives any information via the communication unit 22. The components of the server device 2 are connected to each other so that they can communicate with each other, for example, via a dedicated line.

[0020] The storage unit 23 also stores an image recognition model 24. The control unit 21 constructs the image recognition model 24 that recognizes an object from an image by machine learning using annotation images containing the object as training data. Information for implementing the image recognition model 24 is stored in the storage unit 23. The objects contained in the images used as training data include any products, parts, tools, etc., such as bolts, nuts, washers, iron pipes, and wrenches.

[0021] The operation of the information processing device 1 according to this embodiment will be described with reference to Fig. 2 and Figs. 3A to 3F. Fig. 2 is a flowchart showing an example of the operation of the information processing device 1. Fig. 3A is a diagram showing an example of a captured image. Fig. 3B is a diagram showing an example of a recognition result image. Figs. 3C to 3F are diagrams showing examples of user interfaces provided by the information processing device 1.

[0022] In step S1 of Fig. 2, the control unit 11 of the information processing device 1 acquires a captured image obtained by capturing an image of a recognition target. The captured image is generated by the user capturing an image of a recognition target placed at the user's work base using the imaging unit 16 of the information processing device 1 or an imaging device including a digital camera. Fig. 3A shows an example of a captured image. The captured image G1 shown here includes a recognition target 31. In this case, the recognition target 31 is nine T-shaped bolts, two circular washers, and five hexagonal nuts.

[0023] 2, the control unit 11 of the information processing device 1 acquires the recognition result. The control unit 11 transmits the captured image together with a recognition request to the server device 2 in response to a user operation. In the server device 2, the control unit 21 recognizes recognition objects from the captured image using the image recognition model 24 and counts the number of recognition objects. The control unit 11 receives the recognition result obtained from the captured image by the image recognition model 24 from the server device 2. The recognition result includes information on the type of recognition object and information on the number of each type of recognition object.

[0024] In step S3 of FIG. 2, the control unit 11 of the information processing device 1 displays an image of the recognition result on the display unit 14. FIG. 3B shows an example of the recognition result image. In the recognition result image G2 shown here, the recognition objects 31 are displayed in different display modes for each type. The display mode may be a color, a hatching pattern, or the like. Alternatively, the display mode may be a mode in which a translucent mask image is superimposed on the recognition objects 31. The types of the recognition objects 31 are, for example, T-shaped bolts, circular washers, and hexagonal nuts. Furthermore, the recognition result image G2 indicates information on the number of each type of recognition object 31. The number information is displayed as a serial number for each type of recognition object 31 superimposed on the corresponding recognition object 31. The recognition result image G2 includes unrecognized objects 32 that have not been recognized by the image recognition model 24. The unrecognized objects 32 are, for example, two T-shaped bolts and one hexagonal nut that were not recognized because they overlap with other recognition objects 31. The unrecognized object 32 is not displayed in the same display manner as the recognition object 31 of the same type, and no serial number is displayed. Furthermore, the recognition result image G2 includes a second object (hereinafter referred to as a misrecognized object 33) that is a part of the background that has been mistakenly recognized as the recognition object 31. The misrecognized object 33 is, for example, a circular object that appears in the background. Such a circle is displayed in the same display manner as a circular washer that is part of the recognition object 31, and the serial number is displayed superimposed on it. The user can identify the unrecognized object 32 or the misrecognized object 33 by visually checking the recognition result image G2.

[0025] In step S4 of FIG. 2, the control unit 11 receives a user operation for correcting the recognition result via the input unit 15. FIGS. 3C to 3E show examples of the operation for correcting the recognition result. The operation for correcting the recognition result is either or both of an operation for identifying an unrecognized object 32 and adding it as a recognition object 31 (hereinafter referred to as an adding operation) and an operation for identifying a misrecognized object 33 and excluding it from the recognition object 31 (hereinafter referred to as an excluding operation). A series of operations for correcting the recognition result is performed by, for example, a touch operation with the user's finger or the like via the input unit 15. In the example of FIG. 3C, the control unit 11 displays an operation screen 300 on the display unit 14, which includes a recognition result image G2 and a message M1 indicating a menu such as "edit," to prompt the user to make a selection. When the user selects the "Edit" menu, the control unit 11 transitions to the operation screen 301 and displays a message M2, such as "Please select the object to edit," on the display unit 14 to prompt the user to select a type of the recognition object 31, together with an icon I1 indicating the type of the recognition object 31, and accepts the type selection from the user. The operation screen 301 shows an example in which the icon I1 indicating a T-shaped bolt is selected. Next, the control unit 11 transitions to the operation screen 302 and displays a message M3, such as "Do you want to add an unrecognized object 32?", related to an operation to add an unrecognized object 32, and a message M4, such as "Do you want to reduce the number of misdetected objects?", related to an operation to remove a misrecognized object 33, on the display unit 14, and accepts the user's selection. The operation screen 302 shows an example in which the message M3, such as "Do you want to add an unrecognized object 32," is selected.

[0026] FIG. 3D is a diagram illustrating an example of an adding operation. As described with reference to FIG. 3C, when a message M3 related to the adding operation is selected on the operation screen 302, the control unit 11 transitions to the operation screen 303 and displays a message M5 on the display unit 14, such as "Please surround the object to be added," prompting the user to identify the unrecognized object 32. The control unit 11 accepts an operation to identify the unrecognized object 32 from the user. The user operation to identify the unrecognized object 32 is, for example, an arbitrary operation to identify a closed region including the unrecognized object 32 by a touch operation or the like. In this example, the unrecognized object 32 is identified as, for example, a T-shaped bolt. When the control unit 11 accepts the identification of the unrecognized object 32 from the user, the control unit 11 transitions to the operation screen 304 and displays a message M6, such as "Complete," on the display unit 14 to indicate the end of the identifying operation, prompting the user to make a selection. On the operation screen 301 in FIG. 3C , a T-shaped bolt is selected as the type of the recognition target object 31, and therefore, on the image G3 for the specific operation on the operation screen 304, the unrecognized object 32 is displayed as the identified T-shaped bolt in a display mode different from that of the other recognition targets 31. For example, the identified unrecognized object 32 is displayed with a brightness or saturation higher than the brightness or saturation used to display the other recognition targets 31. In addition, the control unit 11 adds an annotation to the recognition result image to add the identified unrecognized object 32 as a T-shaped bolt, which is one of the recognition targets 31.

[0027] FIG. 3E is a diagram illustrating an exclusion operation. As described with reference to FIG. 3C , when a message M4 related to the exclusion operation of the misidentified object 33 is selected on the operation screen 302, the control unit 11 transitions to an operation screen 305 and displays a message M7 on the display unit 14 prompting the user to identify the misidentified object 33, such as "Touch the object to be removed." In this case, a circular washer is selected as the type of the recognition object 31 to be edited on the operation screen 301 of FIG. 3C . The control unit 11 accepts an operation to identify the misidentified object 33 from the user. The user operation to identify the misidentified object 33 is any operation to identify a closed area including the misidentified object 33 by a touch operation or the like. Here, for example, the misidentified object 33 is identified as a circle that is part of the background. When the control unit 11 accepts the identification of the misidentified object 33 from the user, the control unit 11 displays a message M6 related to the end of the identification operation, such as "Complete," on the display unit 14, as shown on the operation screen 304 of FIG. 3D , to prompt the user to make a selection. 3E, a misrecognized object 33 that has been mistakenly recognized as a circular washer is identified. The control unit 11 adds an annotation to the recognition result image to exclude the identified misrecognized object 33 from a circular washer, which is one of the recognition objects 31.

[0028] The operation of identifying a closed region including the unrecognized object 32 or the misidentified object 33 is performed by tracing or long-pressing the surface of the display unit 14, which constitutes a touch panel display, via the input unit 15, which includes a touch sensor. A specific example of the operation of identifying a closed region is an operation of correcting at least a portion of the contour of the unrecognized object 32. In this case, the operation may be an operation of tracing at least a portion of the contour of the unrecognized object 32. Another example of the operation is an operation of filling in at least a portion of the interior of the unrecognized object 32. Yet another example of the operation is an operation of surrounding the unrecognized object 32 with a polygon. In this case, the operation may be an operation of surrounding the periphery of the unrecognized object 32 with one or more points. Yet another example of the operation is an operation of long-pressing multiple points (e.g., three or more points) on the contour of the unrecognized object 32. In this case, the control unit 11 can surround the unrecognized object 32 by executing an arbitrary image discrimination model on an arbitrary server device, including the server device 2, or on the control unit 11, and inferring other points on the contour of the unrecognized object 32. Yet another example of an operation is an operation of long pressing on the vicinity of the unrecognized object 32. In this case, the control unit 11 can surround the unrecognized object 32 that is closest to the contact position pressed by the user via the input unit 15. Yet another example of an operation is an operation of pressing a point near the unrecognized object 32 to display a closed area (for example, a bounding box) and surrounding the unrecognized object 32 with the closed area. Yet another example of an operation is an operation of long pressing on the closed area including the misrecognized object 33 or tracing the periphery.

[0029] 2, the control unit 11 transmits correction information indicating the corrected recognition result to the server device 2 to correct the image recognition model 24 using the correction information. The correction information includes either or both of information on the unrecognized object 32 and information on the misrecognized object 33. The information on the unrecognized object 32 includes information identifying the unrecognized object 32 as the recognition object 31, that is, a recognition result image annotated to add the unrecognized object 32 to the recognition object 31. The information on the misrecognized object 33 includes information identifying the misrecognized object 33 as not the recognition object 31, that is, a recognition result image annotated to exclude the misrecognized object 33 from the recognition object 31.

[0030] FIG. 3F shows an example of a user interface for transmitting correction information to the server device 2. When the control unit 11 receives a correction of the recognition result from the user as described with reference to FIGS. 3C to 3E, the control unit 11 displays a correction image indicating the corrected recognition result (i.e., correction information) on the display unit 14. The operation screen 306 shown in FIG. 3F includes a correction image G5. In the correction image G5, two T-shaped bolts and one hexagonal nut have been added as unrecognized objects 32 by the user, and therefore, they are displayed in a display manner different from the recognition objects 31 of the same type. On the other hand, in FIG. 3E, a single circle has been identified and excluded as a misrecognized object 33 for a circular washer by the user, and therefore, the excluded misrecognized object 33 is not displayed in the correction image G5. Alternatively, the excluded misrecognized object 33 may be displayed in the correction image G5 in a display manner different from the recognition objects 31 or the misrecognized objects 33, such as a dotted line. At this time, the control unit 11 displays a message M8 on the display unit 14, indicating count information before (i.e., the recognition result) and after (i.e., the corrected recognition result, i.e., correction information) the user has made the correction. An example of the message M8 on the operation screen 306 shows a comparison between the number of each type of recognition target object 31 in the recognition result and the number of each type after the correction. Specifically, the message M8 indicates that the number of T-shaped bolts has increased from 7 to 9, the number of circular washers has decreased from 3 to 2, and the number of hexagonal nuts has increased from 4 to 5. The control unit 11 transitions to the operation screen 307, and displays a message M9 on the display unit 14 prompting the user to finish the correction, such as "Please press Finish if you are satisfied," and accepts a selection such as "Finish" from the user. Upon accepting a selection such as "Finish" from the user, the control unit 11 transmits the correction information to the server device 2. Alternatively, the control unit 11 may transmit the correction information to the server device 2 based on an arbitrary instruction operation by the user. When the control unit 11 receives a selection such as "Complete" from the user, the control unit 11 transitions to the operation screen 308 and displays a message M10 indicating the corrected count information related to the recognition object 31 on the display unit 14. Specifically, the message M10 indicates that there are nine T-shaped bolts, two circular washers, and five hexagonal nuts. At this time, information on the number of each type of recognition object 31 is displayed superimposed on the recognition object 31.Also, the recognition objects 31 are displayed in different colors depending on the type of the recognition objects 31.

[0031] When the server device 2 receives the annotated recognition result image from the information processing device 1, the server device 2 can correct the image recognition model 24 using the annotated recognition result image as training data. The image recognition model 24 is corrected, for example, every time correction information is received from the information processing device 1 (e.g., in real time) or at a predetermined interval. The predetermined interval includes, for example, every predetermined number of times the server device 2 receives correction information, or daily or weekly. Even if the unrecognized object 32 overlaps with the recognition object 31 and the entire appearance of the unrecognized object 32 cannot be confirmed, as shown in the recognition result image G2, correcting the image recognition model 24 using correction information including information about the unrecognized object 32 enables the image recognition model 24 to recognize the recognition object 31, even if only a portion of the appearance is confirmed, as the recognition object 31. Furthermore, even if a portion of the background is erroneously recognized, correcting the image recognition model 24 using correction information including information about the misrecognized object 33 makes it possible to exclude similar background objects from recognition and prevent erroneous recognition, as shown in the recognition result image G2.

[0032] 4A is a diagram showing a modified example of the recognition result image G2. In the example of the recognition result image G6 shown in FIG. 4A, the control unit 11 acquires information on the accuracy with which the image recognition model 24 recognized the recognition target object 31 from the captured image from the server device 2, and displays the information on the recognition result image. The information on the accuracy of the recognition result is a numerical value derived using a softmax function and representing the probability that the image recognition model 24 will recognize the target object. In other words, the higher the numerical value representing the probability, the more likely the recognition target object 31 will be recognized correctly, and the lower the numerical value, the higher the probability that there will be a relative increase in unrecognized or erroneous recognition. The control unit 11 can classify the probability numerical value into any level and display the numerical value for a predetermined level on the recognition result image. For example, when the maximum accuracy of the recognition result is 1, the control unit 11 classifies the recognition targets 31 into three categories: recognition targets 31 recognized with an accuracy within a range of 0.8 to 1 (hereinafter referred to as high recognition targets); recognition targets 31 recognized with an accuracy within a range of 0.6 to 0.8, including an arbitrary reference value of 0.7 (hereinafter referred to as medium recognition targets); and recognition targets 31 recognized with an accuracy within a range of 0 to 0.6 (hereinafter referred to as low recognition targets). Accuracy within a predetermined range including the reference value is an accuracy level that requires the user to determine whether or not the recognition needs to be corrected. Classifying the accuracy within a predetermined range including the reference value makes it possible to prompt the user to check medium recognition targets that require attention. The control unit 11 displays the numerical value of the accuracy of one or more of the high recognition targets, medium recognition targets, and low recognition targets in the recognition result image G6. The numerical values ​​of the accuracy of the high recognition targets and medium recognition targets are displayed in the recognition result image G6. In the recognition result image G6, the accuracy value is displayed by pointing an arrow at the corresponding high- or medium-recognition object, but the accuracy value may also be displayed superimposed on the corresponding high- or medium-recognition object. The accuracy value may also be displayed in a different display mode for each predetermined level. As shown in the example of the recognition result image G6, the high-recognition object, medium-recognition object, and low-recognition object are highly likely to correspond to a correctly recognized recognition object 31, an unrecognized object 32 or a misrecognized object 33, and the background 34, respectively.Therefore, the user can correct the recognition result or identify either or both of the unrecognized object 32 and the misrecognized object 33 while referring to the accuracy obtained from the image recognition model 24. Therefore, when correcting an unrecognized or misrecognized result, the user can make the correction with higher reliability.

[0033] FIG. 4B is a diagram showing an image of a modified example of the recognition result image G6. In the example of the recognition result image G7 shown in FIG. 4B, the control unit 11 changes the display mode in the recognition result image of the recognition target object 31 recognized with a predetermined accuracy. For example, as shown in the example of the recognition result image G7, the control unit 11 displays the medium-level recognition target object in a different display mode so as to make it stand out more than the high-level recognition target object. Therefore, in the example of the recognition result image G7, the unrecognized target object 32 and the misrecognized target object 33 are displayed with a brightness or saturation higher than the brightness or saturation used to display the recognition target object 31. This configuration makes it easier for the user to intuitively grasp the unrecognized target object 32 or the misrecognized target object 33 that will likely need to be corrected. Alternatively, as a modified example, the control unit 11 may control the recognition result image so as not to accept user operations to correct areas other than those showing the medium-level recognition target object.

[0034] FIG. 4C is a diagram showing a modified example of the recognition result image G6. The recognition result image G8 shown in FIG. 4C differs from the recognition result image G7 of FIG. 4B in that the high-level recognition object and the low-level recognition object are displayed in the same manner in order to make the medium-level recognition object stand out. That is, in the example of the recognition result image G8, the unrecognized object 32 or the misrecognized object 33 is displayed with a brightness or saturation higher than the brightness or saturation used to display the recognition object 31 and the background 34. Furthermore, the brightness or saturation used to display the recognition object 31 and the background 34 is the same. This configuration makes it easier for the user to intuitively grasp the unrecognized object 32 or the misrecognized object 33 that will likely need to be corrected.

[0035] Although the present disclosure will be described based on various drawings and examples, it should be noted that those skilled in the art may make various modifications and alterations based on the present disclosure. Other modifications are possible within the scope of the present disclosure. For example, the functions included in each means or step may be rearranged so as not to cause logical inconsistencies, and multiple means or steps may be combined or divided into one.

[0036] The drawings illustrating the embodiments of the present disclosure are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0037] As a variation, the information processing device 1 may be a computer, or may be configured by multiple computers such as a cloud computing system. Alternatively, the server device 2 may be a server belonging to a cloud computing system or other computing system. The server device 2 is installed, for example, in a facility dedicated to a business operator or in a shared facility including a data center. As a further variation, the server device 2 may be mounted on the information processing device 1.

[0038] As a modified example, when the control unit 11 displays a message M6 such as "Complete" relating to the end of the specific operation on the display unit 14 and accepts the user's selection, it displays a message such as "Please select another object to edit" urging the user to continue the correction on the display unit 14 together with an icon I1 indicating the type of the recognition object 31. When the control unit 11 accepts the type of the recognition object 31 to be corrected next from the user, it can continue the operation to correct the recognition result (i.e., the adding operation or the removing operation) as described in Figs. 3C to 3E.

[0039] As another modification, the display positions of the recognition result image G2, images G3 to G4, corrected image G5, and messages M1 to M10 may be in any area of ​​the display unit 14 different from the examples shown in FIGS. 3C to 3F.

[0040] As another modified example, the control unit 11 may switchably display the recognition result image G2, the images G3 to G4, the corrected image G5, and the recognition result images G6 to G8. When the control unit 11 receives switching control from the user via the input unit 15, it can display the recognition result image G2, the images G3 to G4, the corrected image G5, and the recognition result images G6 to G8 in parallel or by switching between them.

[0041] In this embodiment, the control unit 11 transmits correction information including either or both of information identifying the unrecognized object 32 and information identifying the misrecognized object 33 to the server device 2, thereby enabling the server device 2 to use the correction information to correct the image recognition model 24. Therefore, the accuracy of the image recognition model 24 is improved as a result of repeating machine learning using the corrected training data.

[0042] For example, in the above-described embodiment, a program for executing all or part of the functions or processes of the information processing device 1 may be recorded on a computer-readable recording medium. Computer-readable recording media include non-transitory computer-readable media, such as magnetic recording devices, optical discs, magneto-optical recording media, or semiconductor memories. The program may be distributed, for example, by selling, transferring, or lending a portable recording medium, such as a DVD (Digital Versatile Disc) or CD-ROM (Compact Disc Read Only Memory), on which the program is recorded. The program may also be distributed by storing the program in the storage of a server and transmitting the program from the server to another computer. The program may also be provided as a program product. Embodiments of the present disclosure may also be embodied as a system, a program, or a storage medium on which the program is recorded (e.g., an optical disc, a magneto-optical disc, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card).

[0043] The implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module incorporated into an operating system. Furthermore, the program may or may not be configured so that all processing is performed solely by the CPU on the control board. The program may be configured so that part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as needed. [Explanation of symbols]

[0044] 1. Information processing equipment 2. Server device 3 Network 4. Information Processing Systems 11 Control section 12 Communications Department 13 Storage section 14 Display section 15 Input section 16 Imaging unit 21 Control section 22 Communications Department 23 Memory section 24 Image Recognition Model 31 Recognition Object 32 Unrecognized objects 33 Misidentified objects 34 Background 300~308 Operation screen G1 captured image G2 Recognition result image G3~G4 images G5 corrected image G6~G8 Recognition result images I1 Icon M1~M10 Messages

Claims

1. a communication unit that communicates with a server having an image recognition model that recognizes the object from an image, the image being constructed by machine learning using an image including the object as training data; A display unit; an input unit that accepts user operations; a control unit that causes the display unit to display an image of a recognition result obtained by the image recognition model from an image obtained by imaging a first object, receives an operation to correct the recognition result by the input unit, and transmits correction information indicating the corrected recognition result to the server to correct the image recognition model using the training data including the correction information; An information processing device having the above.

2. 2. The information processing device according to claim 1, the operation for correcting the recognition result is an operation for identifying the first object that was not recognized in the captured image, the correction information includes information about the identified first object; Information processing device.

3. 2. The information processing device according to claim 1, the operation for correcting the recognition result is an operation for identifying a second object other than the first object that has been erroneously recognized in the captured image, the correction information includes information about the identified second object; Information processing device.

4. 2. The information processing device according to claim 1, The control unit changes a display mode of the first object recognized with a predetermined accuracy in an image of the recognition result.

5. A program for causing a computer to execute the operations of the information processing device according to any one of claims 1 to 4.

6. a server having an image recognition model that recognizes an object from an image, the image being constructed by machine learning using an image including the object as training data; an information processing device that executes the operation according to any one of claims 1 to 4; A system comprising:

Images