Image processing apparatus and image processing method
By generating and visualizing simulated images that reflect environmental changes, the difficulty of identifying performance evaluation of machine learning models under environmental changes is solved, and visual inspection of the model robustness is realized to ensure the efficient operation of the system in multiple environments.
Patent Information
- Application Number
- CN202380026535.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-11
- Filing Date
- 2023-01-30
- Publication Date
- 2024-11-01
AI Technical Summary
When using machine learning models for image recognition, environmental changes lead to reduced accuracy. The prior art is difficult to effectively evaluate the robustness of the model and cannot visually check the recognition performance of the model.
Through the image processing device and method, simulated images reflecting possible environmental changes are generated, and a recognition status image is superimposed on the simulated images to form visual results, allowing the user to visually check the robustness of the model to environmental changes.
Enables system developers or administrators to visually check the robustness of machine learning models to various possible environment changes, thereby building highly robust systems.
Smart Images

Figure CN118891652A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an image processing device and an image processing method, wherein, when an image recognition operation using a machine learning model is performed to detect a predetermined event from a captured image, the recognition status of the detection target is visualized and presented to a user so that the user can visually check the image recognition performance of the machine learning model. Background Art
[0002] Widely used monitoring systems include systems for detecting predetermined events occurring in a monitoring area by performing image recognition operations on images of the monitoring area captured by a camera. In recent years, the accuracy of image recognition has been significantly improved by using machine learning models constructed using machine learning techniques such as deep learning.
[0003] In the case where a machine learning model is used for image recognition, the machine learning model is a black box; that is, the process in the machine learning model to produce the recognition result is unknown, which means that the user cannot easily check the image recognition performance of the machine learning model. Known technologies for solving this problem in image recognition using machine learning models include technologies for visualizing the basis of the judgment made by the machine learning model to produce the recognition result using images and text (Patent Document 1).
[0004] Prior art literature
[0005] Patent Literature
[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2019-82883 Summary of the invention
[0007] Problem that the invention aims to solve
[0008] In the above-mentioned prior art for image recognition using a machine learning model, images and text are used to visualize and display the basis for the judgments made by the machine learning model to produce image recognition results, which enables users to easily visually inspect the process in the machine learning model for producing image recognition results.
[0009] However, environmental changes may cause various changes in the conditions of the monitored area. In this case, the image recognition operation using the machine learning model may also be affected by environmental changes, which leads to reduced accuracy. Therefore, it is not enough to evaluate the recognition performance of the machine learning model by using images taken under specific conditions, and a technology is needed to evaluate the recognition performance of the machine learning model by using images taken under various conditions caused by environmental changes (that is, a technology that enables system developers or system administrators to check whether the image recognition using the machine learning model is robust enough for various possible environmental changes on site).
[0010] The present disclosure is made in view of the problems of the prior art, and the main purpose of the present disclosure is to provide an image processing device and an image processing method, which enable system developers or system administrators to easily visually check the robustness of the machine learning model against various possible environmental changes on site, so as to thereby build a system with high robustness.
[0011] Solutions for solving problems
[0012] One aspect of the present disclosure provides an image processing device for performing a processing operation, wherein the processing operation is used to visualize the recognition status of a detection target when an image recognition operation is performed using a machine learning model to detect a predetermined event from a captured image, wherein the processing operation is performed by a processor, and the processing operation includes: in response to a user's operation for specifying image processing conditions, performing an image processing operation on an original image based on the specified image processing conditions, thereby generating a simulated image for reproducing an image captured under specific circumstances; and generating a status image representing the recognition status of the detection target when the image recognition operation is performed on the simulated image, and superimposing the generated status image on the simulated image to produce a result image as a visualization result, wherein the result image is output as display information.
[0013] Another aspect of the present disclosure provides an image processing method for performing a processing operation, wherein the processing operation is used to visualize the recognition status of a detection target when an image recognition operation is performed using a machine learning model to detect a predetermined event from a captured image, wherein the processing operation is performed by an information processing device, and the processing operation includes: in response to a user's operation for specifying image processing conditions, performing an image processing operation on an original image based on the specified image processing conditions, thereby generating a simulated image for reproducing an image captured under specific circumstances; and generating a status image representing the recognition status of the detection target when the image recognition operation is performed on the simulated image, and superimposing the generated status image on the simulated image to produce a result image as a visualization result, wherein the result image is output as display information.
[0014] Effects of the Invention
[0015] According to the present disclosure, a simulation image reflecting various possible environmental changes of a scene is generated using an image processing operation based on an image processing condition specified by a user. When an image recognition operation is performed on the simulation image, a condition image representing the recognition condition of a detection target is generated, and the condition image is superimposed on the simulation image to produce a result image as a visualization result, which is output as display information. This configuration enables a system developer or system administrator to easily visually check the robustness of a machine learning model against various possible environmental changes of a scene, thereby constructing a system with high robustness. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 is a diagram showing the overall structure of a robustness check system according to an embodiment of the present disclosure;
[0017] Figure 2 is an explanatory diagram showing an overview of operations performed by the image processing apparatus;
[0018] Figure 3 is a block diagram showing a schematic structure of an image processing apparatus;
[0019] Figure 4 is a block diagram showing an overview of operations performed by an image processing apparatus;
[0020] Figure 5 is an explanatory diagram showing an original image setting screen;
[0021] Figure 6 is an explanatory diagram showing a detection target setting screen;
[0022] Figure 7 is an explanatory diagram showing a processing condition setting screen;
[0023] Figure 8 is an explanatory diagram showing a simulated image display screen;
[0024] Fig. 9 is an explanatory diagram showing a visualization result screen;
[0025] Fig.10 is an explanatory diagram showing a visualization result screen;
[0026] Fig.11 is an explanatory diagram showing another example of a visualization result screen;
[0027] Fig.12 is an explanatory diagram showing a process for calculating a validity score;
[0028] Fig.13 is an explanatory diagram showing a real-time visualization result screen;
[0029] Fig.14 is an explanatory diagram showing a visualization result screen when a plurality of detection targets are specified;
[0030] Fig.15 is an explanatory diagram showing a visualization result screen of a second example in a case where a plurality of detection targets are specified;
[0031] Fig.16 is an explanatory diagram showing another visualization result screen of the second example in a case where a plurality of detection targets are specified; and
[0032] Fig.17 is a flowchart showing the procedure of the operation of the image processing apparatus. DETAILED DESCRIPTION
[0033] The first aspect of the present disclosure made to achieve the above-mentioned purpose is an image processing device for performing a processing operation, wherein the processing operation is used to visualize the recognition status of a detection target when an image recognition operation is performed using a machine learning model to detect a predetermined event from a captured image, wherein the processing operation is performed by a processor, and the processing operation includes: in response to a user's operation for specifying image processing conditions, performing an image processing operation on an original image based on the specified image processing conditions, thereby generating a simulated image for reproducing an image captured under specific circumstances; and generating a status image representing the recognition status of the detection target when the image recognition operation is performed on the simulated image, and superimposing the generated status image on the simulated image to produce a result image as a visualization result, wherein the result image is output as display information.
[0034] According to this configuration, a simulation image reflecting various possible environmental changes of a site is generated using an image processing operation based on an image processing condition specified by a user. When an image recognition operation is performed on the simulation image, a condition image representing the recognition condition of the detection target is generated, and the condition image is superimposed on the simulation image to produce a result image as a visualization result, which is output as display information. This configuration enables a system developer or system administrator to easily visually check the robustness of a machine learning model against various possible environmental changes of a site, thereby constructing a system with high robustness.
[0035] A second aspect of the present disclosure is an image processing device of the first aspect, wherein the processing operations performed by the processor include: presenting a detection target setting screen to a user; and setting a detection target specified by the user in response to the user's operation on the detection target setting screen.
[0036] This configuration enables the user to check the recognition status of various detection targets by changing the detection target designated from the various types of detection targets. The detection target may be an object of a specific type or a specific state of an object of a specific type.
[0037] The third aspect of the present disclosure is the image processing device of the first aspect, wherein the processing operations performed by the processor include: presenting a processing condition setting screen to the user; and setting the image processing conditions specified by the user in response to the user's operation on the processing condition setting screen.
[0038] This configuration enables the generation of simulated images that reproduce images captured under various environmental conditions that may result in an expected loss of accuracy in the monitored area, thereby ensuring that the user can check the robustness of the machine learning model to environmental changes.
[0039] A fourth aspect of the present disclosure is the image processing device of the first aspect, wherein the image processing operation includes at least one of a blur operation, an illumination adjustment operation, and a virtual object superimposition operation.
[0040] In this configuration, the blur operation enables the generation of a simulated image that reproduces, for example, an image captured with a fogged camera lens or an image captured outdoors in fog. The illumination adjustment operation enables the generation of a simulated image that reproduces, for example, an image captured under strong sunlight or an image captured under low sunlight with the lighting device not turned on. The virtual object superimposition operation enables the generation of a simulated image that represents a monitoring area crowded with people or a monitoring area where an object that is a detection target is hidden by other objects.
[0041] The fifth aspect of the present disclosure is an image processing device of the first aspect, wherein the processing operations performed by the processor include: generating a hue image, in which the hue at each part of the simulated image represents a degree of contribution, and the degree of contribution is the degree to which each part accounts for the recognition result of the image recognition operation; and superimposing the hue image as the condition image on the simulated image.
[0042] This configuration enables the user to appropriately grasp the recognition status of the detection target in the image recognition operation. The tone image can be an image in which the tone (hue) at each part of the object image gradually changes according to the "contribution degree" (which is the degree to which each part accounts for the recognition result indicating the object detected in the image recognition operation), or the tone image can be a monochrome image in which the brightness (density) level at each part of the object image gradually changes according to the contribution degree of each part.
[0043] The sixth aspect of the present disclosure is an image processing device of the first aspect, wherein the processing operations performed by the processor include: generating a score image for indicating a score, wherein the score numerically represents the accuracy of the recognition status of the detection target in the simulation image; and superimposing the score image as the status image on the simulation image.
[0044] This configuration enables the user to easily grasp the accuracy of the recognition status of the detection target on the simulated image (i.e., the effectiveness of the machine learning model for the simulated image). The accuracy of the recognition status of the detection target on the simulated image can be quantified, for example, based on the degree of consistency between the area of the detection target in the simulated image and the area of the status image superimposed on the simulated image.
[0045] The seventh aspect of the present disclosure is the image processing device of the first aspect, wherein, when a user specifies multiple detection targets, the processing operations performed by the processor include: generating a status image for each of the multiple detection targets, so that each status image for the multiple detection targets is shown in a visually distinguishable manner; and superimposing the status image on the simulation image.
[0046] This configuration enables the user to visually grasp the recognition status of each detection target. In this case, for example, the status images for the detection targets can be displayed simultaneously in different forms (specifically, in different colors or patterns). In response to the user's operation on the screen to select the type of the detection target by using the selection label displayed on the screen, (one or more than one) detection targets (for which the status images are superimposed on the simulation image) can be changed.
[0047] An eighth aspect of the present disclosure is an image processing method for performing a processing operation, the processing operation being used to visualize the recognition status of a detection target when performing an image recognition operation that uses a machine learning model to detect a predetermined event from a captured image, wherein the processing operation is performed by an information processing device, and the processing operation includes: in response to a user's operation for specifying image processing conditions, performing an image processing operation on an original image based on the specified image processing conditions, thereby generating a simulated image for reproducing an image captured under specific circumstances; and generating a status image representing the recognition status of the detection target when the image recognition operation is performed on the simulated image, and superimposing the generated status image on the simulated image to produce a result image as a visualization result, wherein the result image is output as display information.
[0048] This configuration enables a system developer or system administrator to easily visually check the robustness of a machine learning model against various possible environmental changes in the field in the same manner as the first aspect, to thereby build a system with high robustness.
[0049] Embodiments of the present disclosure will be described below with reference to the accompanying drawings.
[0050] Figure 1 is a diagram showing the overall structure of a robustness check system according to an embodiment of the present disclosure.
[0051] The system includes an image processing device 1 (information processing device), a camera 2 , and a recorder 3 .
[0052] The camera 2 captures images of the monitoring area. The recorder 3 stores the images captured by the camera 2. The image processing device 1 receives the captured images in real time from the camera 2. The image processing device 1 also receives the captured images stored in the recorder 3.
[0053] The image processing apparatus 1 is mainly composed of a personal computer (PC) or the like. A display 4 and an input device 5 such as a keyboard and a mouse are connected to the image processing apparatus 1. The display 4 and the input device 5 may be integrally formed as a touch panel display.
[0054] The image processing device 1 performs processing operations for visualizing the recognition status of a detection target in the case of performing an image recognition operation using a machine learning model to detect a predetermined event from a captured image. The visualization result is presented to a user so that the user can visually check the effectiveness of the machine learning model. In the present embodiment, the image processing device 1 is configured to evaluate the recognition performance of the machine learning model using images captured under various conditions caused by environmental changes; that is, to allow the user to check whether the image recognition operation using the machine learning model is sufficiently robust against various possible environmental changes.
[0055] Next, the processing operation performed by the image processing apparatus 1 will be described. Figure 2 This is an explanatory diagram showing an overview of operations performed by the image processing apparatus.
[0056] The image processing device 1 acquires an original image 21 (original captured image) from the camera 2 or the recorder 3. In this example, the monitoring area (the area captured by the camera 2) is an elevator hall (a space used by people to enter and exit the elevator). In addition, in this example, the detection target is a wheelchair, and the original image 21 shows that the person leaves the elevator while moving the wheelchair.
[0057] The image processing device 1 performs an image processing operation on the original image 21 to generate a simulated image that reproduces an image captured under a specific situation. In response to a user's operation to specify an image processing condition, the image processing device 1 performs the image processing operation according to the specified image processing condition. In other words, based on possible environmental changes in the monitoring area, the user specifies specific image processing conditions (i.e., a set of various parameters for the image processing operation).
[0058] In this embodiment, the image processing operations performed by the image processing device 1 include a blurring operation, an illumination adjustment operation, and a virtual object superimposition operation.
[0059] In the blur operation, a blur image transformation is applied to the original image 21 to generate a simulated image 22 that reproduces a blurred captured image. Specifically, the image processing device 1 generates a simulated image 22 that reproduces an image captured with a fogged camera lens or an image captured outdoors in fog.
[0060] In the illumination adjustment operation, an image transformation that changes the brightness is applied to the original image 21 to generate a simulated image that reproduces an image captured under low illumination conditions and high illumination conditions. Specifically, when the illumination is set to high, the image processing device 1 generates a simulated image that reproduces an image captured under strong sunlight. Conversely, when the illumination is set to low, the image processing device 1 generates a simulated image that reproduces an image captured under low sunlight and in a state where the lighting device is not turned on.
[0061] In the virtual object superimposing operation, a predetermined virtual object image 23 is superimposed on the original image 21. In this example, an image of a person is superimposed on the original image 21 as the virtual object image 23. In the case where the virtual object image 23 is an image of a person, a simulated image is generated to represent a situation in which the monitoring area is crowded with people (there are multiple people in the monitoring area), or a situation in which the object as the detection target is hidden by other objects. The image processing device 1 creates the virtual object image 23 by cutting out a region of an object (such as a person, etc.) from an image previously captured by the camera 2. In some cases, the virtual object image 23 can be generated by using computer graphics (CG). The virtual object image 23 may be a contour image.
[0062] In the present embodiment, the image processing device 1 performs a blur operation, an illumination adjustment operation, and a virtual object superimposition operation as image processing operations. In some cases, the image processing device 1 performs any other operation other than the above three operations as image processing operations. For example, the image processing device 1 may perform an operation to change the resolution of an image. In addition, application software for image editing may be activated so that the image processing device 1 performs various image processing operations in response to user operations on the screen.
[0063] When the image processing device 1 performs an image recognition operation using a machine learning model on the object image (original image 21, simulated image 22), a visualization operation is performed to visualize the status of the detection target. In this embodiment, the image processing device 1 generates visualization results 25, 26 (result images), in which a heat map 27 (status image) indicating the recognition status of the detection target is superimposed on the object image (original image 21, simulated image 22).
[0064] The heat map 27 is a tone image in which the tone (hue) at each part (pixel unit or block unit including a plurality of pixels) of the object image (original image 21, simulated image 22) represents the degree to which each part contributes to the recognition result indicating the object detected in the image recognition operation (hereinafter also referred to as the "contribution degree" of each part). More specifically, the heat map 27 is a tone image in which the tone (hue) at each part of the object image gradually changes according to the contribution degree of the part. For example, the tone (hue) changes in the order of red, yellow, green, and blue with decreasing contribution degree. In other cases, the heat map 27 may be a monochrome image in which the brightness (concentration) level at each part of the object image represents the contribution degree of the part.
[0065] By comparing the image of the object as the detection target in the object image (original image 21, simulated image 22) with the heat map 27 superimposed on the object image, the user can visually check the degree of overlap (consistency) between the two images. More specifically, the user can visually determine whether the area with a high contribution degree in the heat map 27 is located at the center of the image of the object (i.e., the detection target), thereby checking the accuracy of the recognition result of the image recognition operation (i.e., the effectiveness of the machine learning model).
[0066] In the present embodiment, when the image recognition operation using the machine learning model is performed on the object image (original image 21, simulated image 22), the condition of the detection target is visualized using the heat map 27 superimposed on the object image (original image 21, simulated image 22). However, the visual representation used for visualization is not limited to the heat map 27.
[0067] Next, the image processing apparatus 1 will be described. Figure 3 is a block diagram showing a schematic structure of the image processing apparatus 1, and Figure 4 It is a block diagram showing an overview of operations performed by the image processing device 1 .
[0068] The image processing device 1 includes a communication device 11 , a storage unit 12 , and a processor 13 .
[0069] The communication device 11 communicates with the camera 2 and the recorder 3 .
[0070] The storage unit 12 stores programs executable by the processor 13 and other data.
[0071] The processor 13 performs various processing operations by executing the program stored in the storage unit 12. In the present embodiment, the processor 13 performs an original image acquisition operation, a detection target setting operation, an image processing condition setting operation, an image processing operation, an image recognition operation, a judgment basis extraction operation, a visualization operation, an output operation, and other processing operations.
[0072] In the raw image acquisition operation, the processor 13 acquires a captured image (raw image) received from the camera 2 or the recorder 3 through the communication device 11 .
[0073] In the detection target setting operation, in response to the user's operation, the processor 13 sets the type of object as the detection target of the image recognition operation using the machine learning model (the type of object is a condition of the image recognition operation). In this operation, in addition to setting the object type as the detection target, the processor 13 can also set the state of the object (object state) as the condition of the detection target. Specifically, the processor 13 can set a specific object in a specific state (for example, a person in a fallen state) as the detection target.
[0074] In the image processing condition setting operation, the processor 13 sets the image processing conditions (conditions of the image processing operation) according to the user's operation.
[0075] In the image processing operation, the processor 13 processes the original image based on the image processing conditions set in the image processing condition setting operation to generate a simulated image. Specifically, the processor 13 performs a blurring operation, an illumination adjustment operation, and / or a virtual object superimposition operation as the image processing operation (see Figure 2 ).
[0076] In the image recognition operation, the processor 13 recognizes an object as a detection target set in the detection target setting operation from the object image (original image and simulated image) by using a machine learning model (image recognition engine).
[0077] In the judgment basis extraction operation, the processor 13 extracts judgment basis information from the machine learning model used in the image recognition operation. Specifically, the processor 13 extracts the judgment basis information contained in the intermediate layer of the neural network constituting the machine learning model. The judgment basis information is information related to the basis of the judgment made by the machine learning model to generate the recognition result of the image recognition operation for the object image (original image, simulated image) (that is, information indicating the recognition status of the detection target in the image recognition operation for the object image).
[0078] In the visualization operation, the processor 13 generates a heat map that visualizes the judgment basis information, and superimposes the heat map on the object image (original image and simulated image) to generate display information including a visualization result (result image) (see Figure 2 ). In addition to machine learning models used for image recognition operations, other machine learning models can also be used for visualization operations.
[0079] In the output operation, the processor 13 outputs display information to display screens, which include a detection target setting screen ( Figure 6 ), an image processing condition setting screen ( Figure 7 ), and a visualization result screen for presenting visualization results to the user ( Fig. 9 and Fig.10 ).
[0080] Next, the screen displayed on the display 4 will be described. Figure 5 is an explanatory diagram showing an original image setting screen. Figure 6 : is an explanatory diagram showing a detection target setting screen. Figure 7 is an explanatory diagram showing a processing condition setting screen. Figure 8It is an explanatory diagram showing a simulation image display screen. Fig. 9 and Fig.10 It is an explanatory diagram showing a visualization result screen.
[0081] Figure 5 The original image setting screen 101 shown has an original image display section 102. When the user operates the original image display section 102, the display indicates an original image selection screen (not shown). The original image selection screen enables the user to select a file of a captured image stored in the recorder 3. As a result, the image stored in the selected recorder 3 is provided as an original image to the image processing device 1. The original image selection screen also enables the user to select the camera 2. As a result, the real-time image output from the camera 2 is provided as an original image to the image processing device 1. When the original image is input to the image processing device 1, the screen changes to the detection target setting screen 111 ( Figure 6 ).
[0082] exist Figure 6 In the detection target setting screen 111 shown, the original image 21 input to the image processing device 1 is displayed in the original image display unit 102 .
[0083] The detection target setting screen 111 has a detection target designation section 112. When the user operates the detection target designation section 112, a detection target list 113 is displayed, which enables the user to select a detection target from the detection target list. In this example, the user can select a person, a wheelchair, a baby carriage, a bicycle, or any other object as (one or more than one) detection target.
[0084] The detection target setting screen 111 has a "setting" button 114 and a "register" button 115 for registration. When the user operates the "setting" button 114, the screen changes to the image processing condition setting screen 121 ( Figure 7 ).
[0085] Figure 7 The illustrated image processing condition setting screen 121 has image processing condition sections 122 for different types of image processing operations. In this example, the image processing condition sections 122 are provided for three types of image processing operations, namely, blurring operations, illumination adjustment operations, and virtual object superimposition operations.
[0086] The image processing condition section 122 allows the user to specify the image processing condition. Upon receiving the user's specification, the image processing apparatus 1 performs an image processing operation based on the specified image processing condition to display the simulated image 22 subjected to the image processing operation in the corresponding image processing condition section 122. The user can visually check the simulated image 22 to confirm whether an appropriate simulated image 22 is obtained by the image processing based on the specified image processing condition.
[0087] Specifically, the image processing condition section 122 for the blur operation includes a level adjustment section 123. The level adjustment section 123 enables the user to adjust the level (degree) of blur.
[0088] The image processing condition section 122 corresponding to the illuminance adjustment operation includes a level adjustment section 124. The level adjustment section 124 enables the user to adjust the level of illuminance (adjust brightness).
[0089] The image processing condition section 122 corresponding to the virtual object superimposing operation includes an "advanced setting" button 125. When the user operates the "advanced setting" button 125, the display indicates an image editing screen (not shown). The image editing screen enables the user to perform image editing to superimpose a predetermined virtual object image 23 (such as a character image, etc.) on the original image 21.
[0090] The virtual object image 23 may be an image extracted in advance from an image captured by the camera 2. The virtual object image 23 may be an image generated by computer graphics. An image editing screen (not shown) allows the user to operate the screen to adjust the position and size of the virtual object image 23 when the virtual object image 23 is superimposed on the original image 21. In addition, when the virtual object image 23 is generated by CG using a 3D model, the user is allowed to operate the screen to adjust the orientation of the virtual object when the virtual object image 23 is superimposed on the original image 21.
[0091] The user operates the image processing condition section 122 to specify the corresponding image processing condition, which allows the user to visually check the simulated image 22 displayed in the image processing condition section 122 to confirm whether an appropriate simulated image 22 is obtained by image processing based on the specified image processing condition. Upon confirmation, the user operates the "register" button 115, thereby registering the simulated image 22 generated by image processing based on the specified image processing condition, which causes the screen to transition to the simulated image display screen 131 ( Figure 8 ).
[0092] Figure 8 The simulation image display screen 131 shown has a simulation image display section 132. The simulation image display section 132 displays the simulation image 22 generated by image processing based on designated image processing conditions. The simulation image display section 132 includes a scroll bar 135. By manipulating the scroll bar 135, the user can display the portion of the simulation image 22 outside the screen.
[0093] When the user operates the "Set" button 114, the screen returns to the image processing condition setting screen 121 ( Figure 7). Then, the user specifies different image processing conditions and operates the "register" button 115, thereby registering simulated images 22 representing different conditions. By repeating the above-mentioned registration operation, the user can register multiple simulated images representing different conditions. In this example, the simulated images that can be registered are a simulated image 22 representing a blurred condition generated by a blur operation, a simulated image 22 representing a strong or weak sunlight condition (i.e., a high or low illumination condition) generated by an illumination adjustment operation, and a simulated image 22 representing a crowded condition generated by a virtual object superimposition operation.
[0094] The simulation image display screen 131 has a "visualize" button 133. After the registration of the desired simulation image 22 is completed, the user operates the "visualize" button 133 to cause the processor 13 to perform an image recognition operation, a judgment basis extraction operation, and a visualization operation, whereby the screen changes to a visualization result screen 141 (see Fig. 9 ).
[0095] Fig. 9 The visualization result screen 141 shown has a heat map original image portion 142. The heat map original image portion 142 displays a visualization result 25 based on the original image 21. The visualization result 25 is an image in which a heat map 27 is superimposed on the original image 21, and the heat map 27 is a visualization of the basis for the judgment made by the machine learning model to generate the recognition result of the image recognition operation for the original image 21.
[0096] The visualization result screen 141 has a heat map simulated image section 143. The heat map simulated image section 143 displays visualization results 26 based on the simulated image 22. Each visualization result 26 is an image in which a heat map 27 is superimposed on the original image 21, and the heat map 27 is a visualization of the basis of the judgment made by the machine learning model to generate the recognition result of the image recognition operation for the simulated image 22.
[0097] The user may then visually inspect the visualization result 25 displayed in the heat-mapped original image portion 142 and the visualization result 26 displayed in the heat-mapped simulated image portion 143 to determine whether the image recognition operation using the machine learning model is performed appropriately.
[0098] Fig. 9 The visualization result screen 141 shown in FIG. 1 is an example in which robustness to environmental changes is not a problem. Fig. 9 In the visualization result screen 141 in FIG. 2 , the heat-mapped simulation image section 143 shows the visualization results 26 for all the simulation images 22 and indicates appropriate individual heat maps 27 .
[0099] Fig.10The visualization result screen 141 shown is an example of a situation where robustness to environmental changes is a problem. Fig.10 In the visualization result screen 141 in FIG. 1 , the heat map simulated image portion 143 shows that a portion of the visualization result 26 for the simulated image 22 fails to show the heat map 27 (see the result for “illuminance - 20”). In this example, the visualization result 26 of the simulated image 22 that reproduces the image captured under low illumination conditions fails to include the heat map 27. Since the visualization result shows that the recognition accuracy decreases under low illumination conditions, the user can confirm that there is a problem with the recognition accuracy under low illumination conditions. In this case, the user can additionally train the machine learning model using the data of the images captured under low illumination conditions to improve the recognition accuracy under low illumination conditions, thereby improving the robustness of the machine learning model to environmental changes.
[0100] Thus, in this embodiment, simulated images 22 are generated that reproduce images captured under various circumstances, and the recognition status of the detection targets in these simulated images 22 is visualized as a heat map 27, which allows the user to visually check the effectiveness of the machine learning model under various circumstances. When it is confirmed that there is a problem with the recognition accuracy under a specific situation, the user can further use the data of the images captured under the specific situation to train the machine learning model, thereby improving the robustness of the machine learning model against environmental changes.
[0101] Next, another example of the visualization result screen will be described. Fig.11 is an explanatory diagram showing another example of the visualization result screen. Fig.12 is an explanatory diagram showing a process for calculating the effectiveness score.
[0102] exist Fig.11 In the visualization result screen 141 shown, the heat-mapped original image section 142 and the heat-mapped simulated image section 143 display heat-mapped images, each of which includes an effectiveness score image 145 (status image) indicating the effectiveness score for the heat-mapped image. The effectiveness score is a numerical expression of the accuracy of the recognition status of the detection target in the image recognition operation for the object image (original image 21, simulated image 22); that is, the effectiveness score expresses the effectiveness of the machine learning model for the object image. In this example, the score image 145 is superimposed on each of the visualization results 25 and 26 for the object image (original image 21, simulated image 22).
[0103] The visualization result screen 141 has a statistical information section 146. The statistical information section 146 displays statistical information on the effectiveness scores of the simulation images 22. Specifically, the statistical information section 146 displays the average value (average score), the highest value (MAX), and the lowest value (MIN) of the effectiveness scores of all the simulation images 22.
[0104] To calculate the effectiveness score, Fig.12 As shown in (A), the user pre-sets a rectangular frame 31 in the area occupied by the detection target in the original image 21. The rectangular frame 31 is entered by the user as annotation information. Specifically, the user visually determines and specifies the area occupied by the detection target on the original image 21 displayed on the screen.
[0105] When calculating the effectiveness score, if Fig.12 As shown in (B), the processor 13 compares the area of the heat map 27 contained in the visualization results 25 and 26 for the object image (original image 21, simulated image 22) with the area of the detection target (rectangular frame 31), and then calculates the effectiveness score based on the degree of consistency (overlapping rate) between the two areas. For example, when the displacement between the area of the heat map 27 and the area of the detection target (rectangular frame 31) is small (that is, the degree of consistency between the two areas is high), the effectiveness score is high. When calculating the effectiveness score, the processor 13 can reduce the weight of the area in the heat map 27 that has a low contribution to the recognition result.
[0106] Next, the real-time visualization result screen shown in the display 4 will be described. Fig.13 It is an explanatory diagram showing a real-time visualization result screen.
[0107] exist Figure 7 , Figure 8 and Fig. 9 In the example shown, the user operates the image processing condition setting screen 121, specifies the image processing conditions of the image processing operation to generate a simulated image, and then operates the "Visualization" button 133 on the simulated image display screen 131, whereby the screen transitions to the visualization result screen 141, in which the visualization results 25, 26 for the object image (original image 21, simulated image 22) are displayed.
[0108] exist Fig.13 In the example shown, in response to a user's operation to specify image processing conditions of an image processing operation for generating a simulated image, a real-time visualization result screen 151 shows a simulated image 22 and a visualization result 26 that reflect the specified image processing conditions in real time.
[0109] The real-time visualization result screen 151 is connected to the image processing condition setting screen ( Figure 7 ) has an image processing condition section 152 for different types of image processing operations in a similar manner. Specifically, an image processing condition section 152 is provided for a blur operation, an illumination adjustment operation, and a virtual object superimposition operation. The image processing condition section 152 enables a user to specify an image processing condition.
[0110] The real-time visualization result screen 151 has a simulation image display section 153. The simulation image display section 153 displays the simulation image 22 generated by the image processing operation based on the specified image processing conditions so that the displayed simulation image 22 reflects the user's operation on the image processing condition section 152 (if any).
[0111] The real-time visualization result screen 151 has a heat map simulated image section 154. The heat map simulated image section 154 displays the visualization result 26 based on the simulated image 22. Each visualization result 26 is an image in which a heat map 27 is superimposed on a corresponding simulated image in the simulated image 22, and the heat map 27 is a visualization of the basis of the judgment made by the machine learning model to generate the recognition result of the image recognition operation for the simulated image 22.
[0112] Then, the user can visually check the simulated image 22 displayed in the simulated image display section 153 to confirm whether an appropriate simulated image 22 is obtained through image processing based on the specified image processing conditions. At the same time, the user can visually check the visualization result 26 displayed in the heat map simulated image section 154 to determine whether the image recognition operation using the machine learning model is appropriately performed.
[0113] Next, the visualization result screen in the case where a plurality of detection targets are designated will be described. Fig.14 This is an explanatory diagram showing a visualization result screen when a plurality of detection targets are designated.
[0114] Fig.14 The visualization result screen 161 shown allows the user to designate a plurality of detection targets in the detection target designation unit 112 .
[0115] In the visualization result screen 161, the heat-mapped original image section 142 displays the visualization result 25, and the heat-mapped simulated image section 143 displays the visualization result 26. Each visualization result in all the visualization results (i.e., each of the visualization results 25 and the visualization results 26) includes a plurality of detection targets, for which the corresponding heat maps 27 are simultaneously displayed in different forms (specifically, in different colors or patterns). In this example, two types of detection targets (wheelchair, person) are selected, and the heat map for one detection target can be displayed in warm colors and the heat map for another detection target can be displayed in cold colors.
[0116] The visualization result screen 161 has a legend display section 162, and the heat-mapped simulated image section 143 and the heat-mapped original image section 142, which display the heat map 27. The legend display section 162 allows the user to determine which detection target among the detection targets corresponds to each heat map 27.
[0117] In this example, two types of objects (wheelchair and person) are selected as detection targets. However, three or more types of objects may be selected as detection targets.
[0118] Next, a visualization result screen of a second example in the case where a plurality of detection targets are specified will be described. Fig.15 and Fig.16 is an explanatory diagram showing a visualization result screen of a second example in a case where a plurality of detection targets are specified.
[0119] Fig.15 and Fig.16 Visualization result screens 171 each having a tab 172 for selecting one of different types of objects as a detection target are shown. The user can operate the tab 172 to switch the type of object to be detected, so that the visualization result screen displays the visualization result 26 for the selected detection target.
[0120] Fig.15 A case where the user selects a wheelchair by operating the wheelchair label 172 is shown, and the heat map 27 is superimposed on the area of the wheelchair in each simulation image 22 with the corresponding visualization result 26 . Fig.16 Another case is shown where the user selects a character by operating the character tag 172 , and the heat map 27 is superimposed on the region of the character in each simulation image 22 having the corresponding visualization result 26 .
[0121] In a similar manner to the heat-mapped simulated image section 143 , the heat-mapped original image section 142 also displays a heat map 27 as a visualization result 26 in a case where a detection target is selected by the user's operation on the tab 172 .
[0122] In this example, two types of objects (wheelchairs and people) are selected as detection targets. However, three or more types of objects may be selected as detection targets. In this case, the same number of labels 172 as the types of detection targets are provided in the visualization result screen.
[0123] Next, the procedure of the operation of the image processing apparatus 1 will be described. Fig.17 : is a flowchart showing the procedure of the operation of the image processing apparatus 1 .
[0124] The image processing apparatus 1 first acquires an original image from the camera 2 or the recorder 3 (original image acquisition operation) ( ST101 ).
[0125] Next, in response to the user's operation, the image processing device 1 sets the type of object to be a detection target in the image recognition operation using the machine learning model (detection target setting operation) ( ST102 ).
[0126] Next, the image processing apparatus 1 determines whether to continue to register a new simulation image based on the user's operation (ST103). In this step, when the user selects the image processing condition setting screen 121 ( Figure 7 ) on the simulation image display screen 131 ( Figure 8 ) when the “Visualization” button 133 is operated, the image processing device 1 does not continue to register new simulation images; that is, it is determined that the registration of the simulation image is terminated.
[0127] When the new registration of the simulation image is continued (YES in ST103 ), the image processing apparatus 1 sets the image processing conditions (conditions of the image processing operation) in response to the user's operation (image processing condition setting operation) ( ST104 ).
[0128] Next, the image processing apparatus 1 processes the original image based on the image processing conditions set in the image processing condition setting operation (image processing operation) to generate a simulation image (ST105).
[0129] Next, the image processing apparatus 1 registers the simulation image generated by the image processing operation in the simulation image list (simulation image registration operation) (ST106), and then returns to ST103. In step ST106, in response to the user selecting the image processing condition setting screen 121 ( Figure 7 ) by operating the “register” button 115, the image processing apparatus 1 performs a simulated image registration operation.
[0130] In another case, that is, when the registration of the simulated image is terminated ("No" in ST103), the image processing device 1 uses a machine learning model (image recognition engine) to identify the object that is the detection target set in the detection target setting operation from the original image and the simulated image (ST107).
[0131] Next, the image processing device 1 extracts judgment basis information from the machine learning model used in the image recognition operation (judgment basis extraction operation) ( ST108 ).
[0132] Next, the image processing device 1 generates a heat map that visualizes the judgment basis information, and then superimposes the heat map on the original image and the simulated image, thereby generating a result image as a visualization result (visualization operation) (ST109), and the result image is output as display information.
[0133] Although the specific embodiments of the present disclosure are described herein for illustrative purposes, the present disclosure is not limited to these specific embodiments. Various changes, substitutions, additions and omissions may be made to the elements of the embodiments without departing from the scope of the present invention. In addition, the elements and features of different embodiments may be combined with each other to produce another embodiment of the present disclosure.
[0134] Industrial Applicability
[0135] The image processing device and image processing method according to the present disclosure have the effect of enabling a system developer or system administrator to easily visually check the robustness of a machine learning model against various possible environmental changes on site to thereby build a system with high robustness, and are useful as an image processing device and image processing method in which, when performing an image recognition operation using a machine learning model to detect a predetermined event from a captured image, the recognition status of the detection target is visualized and presented to a user so that the user can visually check the image recognition performance of the machine learning model.
[0136] Description of Reference Numerals
[0137] 1 Image processing device
[0138] 2 Camera
[0139] 3. Recorder
[0140] 4 Displays
[0141] 5 Input Devices
[0142] 11 communication device
[0143] 12 Storage
[0144] 13 Processor
[0145] 21 Original Image
[0146] 22 Simulated images
[0147] 23 Virtual Object Image
[0148] 25 Visualization Results
[0149] 26 Visualization Results
[0150] 27 Heatmap (status image)
[0151] 31 Rectangular frame
Claims
1. An image processing device for performing a processing operation for visualizing a recognition status of a detection target in a case where an image recognition operation using a machine learning model to detect a predetermined event from a captured image is performed, wherein the processing operation is performed by a processor, and the processing operation comprises: In response to a user's operation to designate an image processing condition, performing an image processing operation on an original image based on the designated image processing condition, thereby generating a simulated image for reproducing an image captured under a specific situation; as well as A status image representing a recognition status of a detection target when the image recognition operation is performed on the simulation image is generated, and the generated status image is superimposed on the simulation image to produce a result image as a visualization result, wherein the result image is output as display information.
2. The image processing device according to claim 1, wherein: The processing operations performed by the processor include: Presenting a detection target setting screen to the user; and In response to a user's operation on the detection target setting screen, a detection target designated by the user is set.
3. The image processing device according to claim 1, wherein: The processing operations performed by the processor include: presenting a processing condition setting screen to the user; and In response to a user's operation on the processing condition setting screen, the image processing condition designated by the user is set.
4. The image processing device according to claim 1, wherein: The image processing operation includes at least one of a blurring operation, an illumination adjustment operation, and a virtual object overlay operation.
5. The image processing device according to claim 1, wherein: The processing operations performed by the processor include: generating a hue image in which the hue at each portion of the simulated image represents a contribution degree, the contribution degree being a degree to which each portion contributes to a recognition result of the image recognition operation; and The tone image is superimposed on the simulation image as the condition image.
6. The image processing device according to claim 1, wherein: The processing operations performed by the processor include: generating a score image indicating a score that numerically expresses the accuracy of the recognition status of the detection target in the simulation image; and The score image is superimposed on the simulation image as the situation image.
7. The image processing device according to claim 1, wherein: When the user specifies multiple detection targets, the processing operations performed by the processor include: generating a status image for each of the plurality of detection targets, such that each status image for the plurality of detection targets is shown in a visually distinguishable manner; and The situation image is superimposed on the simulation image.
8. An image processing method for performing a processing operation for visualizing a recognition status of a detection target in a case where an image recognition operation using a machine learning model to detect a predetermined event from a captured image is performed, wherein the processing operation is performed by an information processing device, and the processing operation comprises: In response to a user's operation to designate an image processing condition, performing an image processing operation on an original image based on the designated image processing condition, thereby generating a simulated image for reproducing an image captured under a specific situation; as well as A status image representing a recognition status of a detection target when the image recognition operation is performed on the simulation image is generated, and the generated status image is superimposed on the simulation image to produce a result image as a visualization result, wherein the result image is output as display information.
