Information processing apparatus, method of controlling information processing apparatus, program product, and storage medium

Abstract

The invention relates to an information processing apparatus, a method of controlling the information processing apparatus, a program product, and a storage medium. The information processing apparatus includes: an obtaining unit that obtains an image in which a work area and an object are captured; a first detection unit that detects a working area from the image; a second detection unit that detects an object region from the image; a frequency map generation unit that generates a frequency map on the basis of the number of times of detection of the object region of each grid in the working region; a heat map generation unit that generates a heat map in a case where a frequency map is caused to correspond to the working area detected by the first detection unit; and a display control unit that superimposes and displays the heat map at a position based on the working area on the display device.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an information processing apparatus that performs visualization of a work ratio. BACKGROUND

[0002] In a production site, an image processing sensor is used to check whether a work standard is satisfied.

[0003] For example, Japanese Patent Laid-Open No. 2022-157349 describes a work management system that manages an operation including a repetitive operation in which a worker repeatedly moves a finishing instrument. A position of a work element such as the worker or a finger or the finishing instrument is recognized, a motion vector of a movement of the work element is stored, and a quality of the work is managed.

[0004] Depending on a work standard, there is a case where a visualization of a ratio of a work expected to be performed is desired. For example, in a case where a color is applied using a paint or the like, a work ratio can be visualized using color information. Also, even in a case where color information cannot be used, work management can be performed via an image processing sensor according to a technique described in Japanese Patent Laid-Open No. 2015-186202. In the technique described in Japanese Patent Laid-Open No. 2015-186202, from mobile body position information and stay information for each predetermined measurement interval, a heat map for visualization of the stay information of the mobile body is generated.

[0005] Therefore, in a case where a relative position between the image processing sensor and an article does not change or a color or a shape does not change, work management using the image processing sensor is possible.

[0006] However, in a case where a component moved by a belt conveyor is applied with a colorless protective agent or is wiped with alcohol, determination of a work ratio performed using a change in an appearance of a work area, which is performed within the scope of the technique described in the above patent document, cannot be performed.

[0007] Failure to properly obtain a work ratio problematically results in failure to perform determination of whether work is being performed without unevenness or without deviation with respect to a work area. SUMMARY

[0008] The present disclosure is made in view of the above problem and makes it possible to realize an information processing apparatus that can properly visualize a work ratio with respect to a work area.

[0009] According to a first aspect of the present disclosure, there is provided an information processing apparatus including: an obtaining unit that obtains an image that captures a work area and an object; a first detecting unit that detects the work area from the image; a second detecting unit that detects an object area from the image; a frequency map generating unit that generates a frequency map based on a number of detections of the object area per grid in the work area; a heat map generating unit that generates a heat map in a case where the frequency map is made to correspond to the work area detected by the first detecting unit; and a display control unit that superimposely displays the heat map on a display device at a position based on the work area.

[0010] According to a second aspect of the present disclosure, there is provided a method of controlling an information apparatus, the method including: obtaining an image that captures a work area and an object; performing a first detection of detecting the work area from the image; performing a second detection of detecting an object area from the image; generating a frequency map based on a number of detections of the object area per grid in the work area; generating a heat map in a case where the frequency map is made to correspond to the work area detected by the first detection; and superimposely displaying the heat map on a display device at a position based on the work area.

[0011] According to a third aspect of the present disclosure, there is provided a program product for causing a computer to execute the method of controlling the information processing apparatus described above.

[0012] According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing a program for causing a computer to execute the method of controlling the information processing apparatus described above.

[0013] The features of the present disclosure will become apparent from the following description of the embodiments with reference to the accompanying drawings. The following description of embodiments is described by way of example. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the embodiments.

[0015] Figure 1 is a block diagram illustrating an example of a hardware configuration of a recording apparatus according to the first embodiment.

[0016] Figure 2 is a diagram illustrating an example of a schematic configuration of an information processing system including the recording apparatus.

[0017] Figure 3 is a block diagram illustrating an example of a functional configuration of the recording apparatus.

[0018] Figure 4 FIG. 2 is a diagram illustrating an example of a work target region obtained by a work region obtaining unit.

[0019] Figure 5 FIG. 3 is a diagram for describing processing of an object detection unit.

[0020] Figure 6 FIG. 4 is a diagram for describing processing of a region correction unit.

[0021] Figure 7 FIG. 5 is a diagram for describing processing of a heat map generation unit.

[0022] Figure 8 FIG. 6 is a flowchart illustrating a process of generating a heat map.

[0023] Figure 9 FIG. 7 is a diagram for describing processing of a region correction unit according to a second embodiment.

[0024] Figure 10 FIG. 8 is a diagram for describing a correction criterion of a region correction unit.

[0025] Figure 11 FIG. 9 is a diagram illustrating a corrected work completion region.

[0026] Figure 12 FIG. 10 is a diagram for describing processing of a frequency map generation unit.

[0027] Figure 13 FIG. 11 is a diagram for describing processing of frequency information for correcting a frequency map.

[0028] Figure 14 FIG. 12 is a diagram illustrating an example of a heat map display.

[0029] Figure 15 FIG. 13 is a block diagram illustrating an example of a functional configuration of a recording apparatus according to a third embodiment. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claims. In the embodiments, a plurality of features are described, but not all of such features are required, and a plurality of such features can be appropriately combined. Furthermore, in the drawings, the same or similar configurations are given the same reference numerals, and redundant description thereof is omitted.

[0031] First Embodiment

[0032] According to the first embodiment described herein, an information processing apparatus corrects a frequency map indicating a work completion region that has performed work on a work target region obtained from a captured image in accordance with a shape of the work target region, and displays the frequency map as a heat map in superposition.

[0033] In the present embodiment described herein, a process of correcting a heat map display using the vertical and horizontal dimensions of a work target region is executed.

[0034] Figure 1 is a block diagram illustrating an example of a hardware configuration of the recording apparatus 100 according to the present first embodiment.

[0035] The recording apparatus 100 includes a CPU 101, a ROM 102, a RAM 103, a storage apparatus 104, and a communication I / F 105. The CPU 101, the ROM 102, the RAM 103, the storage apparatus 104, and the communication I / F 105 are connected to an internal bus 106.

[0036] The CPU 101 is a central processing unit that comprehensively controls operations in the recording apparatus 100. The ROM 102 is a nonvolatile memory that stores programs required for the CPU 101 to execute processes, various types of initial setting data, and the like. The RAM 103 is a volatile memory that functions as a main memory, a work area, and the like of the CPU 101, and temporarily stores programs, image data, and the like. By the CPU 101 loading required programs and the like from the ROM 102 onto the RAM 103 at the time of executing a process and executing the programs and the like, various types of functional operations are realized.

[0037] The storage apparatus 104 is a storage device having a large capacity compared to the RAM 103, and for example, uses a hard disk drive (HDD) or a solid state drive (SSD). The storage apparatus 104 stores an OS (basic software) that is executed by the CPU 101. Various types of programs for executing a work target region acquisition, a work completion region acquisition, a region correction, a frequency map generation, and a heat map generation according to the present embodiment are also stored; various types of parameters related to the programs; data; and the like. The storage apparatus 104 can also store image data acquired via a network.

[0038] When the CPU 101 is started, for example, by turning on a power supply, the CPU 101 executes a boot program stored in the ROM 102. The boot program is configured to read out an OS stored in the storage apparatus 104 and load the OS onto the RAM 103.

[0039] After the OS is booted, for example, a user can use Figure 2The operation unit 107 shown in the diagram instructs the execution of the target area acquisition program, the completed area acquisition program, the area correction program, the frequency map generation program, and the heat map generation program. At this time, the CPU 101 reads each program from the storage device 104 and loads them into the RAM 103. In this way, the CPU 101 generates a frequency map of the completed area from the corrected completed area. Furthermore, various types of data used in the operation of the program for overlaying heat maps to match the target area of ​​the captured image are also stored and read into the RAM 103.

[0040] The Communication I / F 105 is, for example, a Local Area Network (LAN) interface used for communication with network cameras, another computer, etc., via a network.

[0041] Note that the various types of images used in each of the work completion area acquisition program, area correction program, frequency map generation program, and heat map generation program according to this embodiment, as well as the images used in each program, can be obtained via a network.

[0042] The recording device 100 can be composed of a Figure 1 The diagram illustrates a configuration for a single device (such as a personal computer (PC), tablet PC, etc.). However, Figure 1 The configuration shown in the diagram can be configured by separate hardware blocks. In other words, the recording device 100 can be configured by multiple devices. Moreover, computational processing for various types of analysis and determination can be performed using a GPU (not shown).

[0043] Figure 2 This is a diagram illustrating an example of a schematic configuration of an information processing system 200 including a recording device 100 according to this embodiment.

[0044] The information processing system 200 includes a recording device 100, a storage device 109, and a camera 110.

[0045] The recording device 100, storage device 109 and camera 110 are connected to each other via network 111 in a manner that enables them to communicate.

[0046] Network 111 can be, for example, a LAN. Note that any communication standard, scale, and configuration can be used as long as network 111 is configured to enable communication between recording device 100, camera 110, and storage device 109. Furthermore, the physical connection mode to network 111 can be wired or wireless. Moreover, the connection between recording device 100, storage device 109, and camera 110 is not limited to via network 111, and can also use connections via USB, etc.

[0047] As part of the hardware configuration, in addition Figure 1In addition to the configuration shown in the diagram, the recording device 100 may also include an operation unit 107 and a display unit 108. Here, the operation unit 107 includes a pointing device such as a keyboard or mouse. Moreover, the display unit 108 includes a display such as a liquid crystal display (LCD), and is a display device for viewing, for example, when a user (operator) operates the recording device 100.

[0048] Camera 110 is, for example, a network camera, and is an image capture device with the function of capturing images within its field of view and transmitting the captured images to recording device 100 via network 111. Camera 110 may be a pan-tilt-zoom (PTZ) camera configured to change the imaging area. In this case, camera 110 may include the function of transmitting imaging parameter information, including imaging direction and imaging angle (including pan angle, tilt angle, and zoom ratio), to recording device 100 via network 111.

[0049] Furthermore, in this embodiment described herein, camera 110 is a network camera used as a surveillance camera. However, such limitation is not intended. For example, camera 110 may be a digital still camera, a digital video camera, a smartphone or tablet terminal with camera functionality, a camera for industrial use, a vehicle-mounted camera, a wearable camera, etc. Recording device 100 receives captured images from camera 110 via network 111 as input images and performs the following processing on the received input images.

[0050] Note that camera 110 can transmit captured images to storage device 109, the storage device of another device, etc., via network 111. In this case, recording device 100 can receive captured images stored in storage device 109, the storage device of another device, etc., as input images via network 111. Here, the input image is not limited to the captured image captured by camera 110. For example, the input image can be a partial image corresponding to a portion of the captured image. Moreover, Figure 2 The configuration of the information processing system 200 shown in the diagram is merely an example, and the device can be integrated as in a tablet PC or similar device. In other words, the recording device 100 can provide the functionality of a camera 110.

[0051] The recording device 100 obtains a target area and a completed area from an image acquired via the camera 110. A frequency map is generated from the two areas, and the thermal image display is corrected based on the frequency map and the shape of the target area. The thermal image is then displayed overlaid on the image.

[0052] The target work area refers to the work zone, components, etc., of the object that serves as the target of the work. The completed work area is the area where the worker's hands or the tools used for the work are detected.

[0053] To generate a frequency map, the number of times work is performed in any area of ​​the target work area is counted. The number of times work is performed is calculated by detecting the number of times the worker's hand or the tools used for work are detected. Execution control is used to display the generated heat map on a monitor or the like via display unit 108 so that the user can inspect the heat map.

[0054] Figure 3 This is a block diagram illustrating an example of the functional configuration of the recording device 100 according to this embodiment.

[0055] like Figure 3 As shown, the recording device 100 includes an image acquisition unit 301, a working area acquisition unit 302, an object detection unit 303, an area correction unit 304, a frequency map generation unit 305, a heat map generation unit 306, and an overlay image display unit 307.

[0056] Note that, Figure 3 At least one or more of the functions of each component of the recording device 100 illustrated in the figure can be implemented by the CPU 101 executing a program. However, in Figure 3 Among the components of the recording device 100 illustrated in the figure, at least one or more can operate as dedicated hardware. In this case, the dedicated hardware operates under the control of the CPU 101.

[0057] The following will use Figure 3 Describe the function of the recording device 100.

[0058] The image acquisition unit 301 receives images acquired via the camera 110, etc.

[0059] The working area acquisition unit 302 detects the working target area from the image acquired by the image acquisition unit 301.

[0060] The object detection unit 303 detects a certain object from the image obtained by the image acquisition unit 301. The object detected here can be the hand of a person performing work, tools used in the work, equipment, etc.

[0061] The area correction unit 304 performs shape transformation of the target area and the completed area obtained by the work area acquisition unit 302 and the object detection unit 303 based on the pre-set correction standard stored in the area correction unit 304.

[0062] The frequency map generation unit 305 generates a frequency map based on the work target area and work completion area, which are corrected according to a shape-based correction standard. For example, if a moving part in a belt conveyor is set as the work target area and an image is acquired at a fixed position via the image acquisition unit 301 of the recording device 100, there may be a difference in size between the part detected from an image acquired at one time point and the part detected from images acquired at different time points. This also applies to the case where the area including the hand of a person working on the belt conveyor is set as the work completion area and an image is acquired at a fixed position via the image acquisition unit 301 of the recording device 100. In such a case, by generating a frequency map based on the work target area and work completion area, which are corrected for the size difference of the area including the part or hand, the work ratio can be appropriately obtained.

[0063] The heatmap generation unit 306 generates a heatmap by correcting the shape of the frequency map generated by the frequency map generation unit 305 to match the working target region in the image obtained by the image acquisition unit 301. In this way, when generating the heatmap, the shape of the frequency map is corrected to match the working target region in the image obtained by the image acquisition unit 301, rather than the working target region whose shape is corrected based on a correction standard. Therefore, a heatmap of the working target region that the user is viewing in real time can be generated.

[0064] The overlay image display unit 307 displays the heat map generated by the heat map generation unit 306 overlaid on the image at the location of the target working area. For example, when a moving part in a belt conveyor is set as the target working area, the generated heat map is overlaid on the image to match the movement of the target working area. In other words, the generated heat map is displayed dynamically to match the movement of the target working area. Therefore, even in the case of a moving target working area, the user can appropriately obtain the working ratio for that target working area.

[0065] The following will use Figure 4 , Figure 5 , Figure 6 and Figure 7 The working area acquisition unit 302, object detection unit 303, area correction unit 304, frequency map generation unit 305, and heat map generation unit 306, which correspond to the functions of the recording device 100, are described in detail.

[0066] The target work area obtained by the work area acquisition unit 302 is, for example, a region of a component or indicator work area corresponding to the target work area. Feature points of the component or indicator work area are extracted, and the region enclosed by the feature points is set as the target work area. Feature points can be extracted using Scale Invariant Feature Transform (SIFT), and corner portions of the component or region corresponding to the target can be extracted. Moreover, different methods can be used for feature point extraction, and the type is not limited.

[0067] Furthermore, markers such as 2D barcodes can be used to obtain the target work area. In this case, the marker is pre-attached to the part or area corresponding to the target work, and coordinate points indicating the extent of the area are obtained by detecting the marker. The method for detecting the area is not limited to this method, and methods including detecting areas via deep learning can be used, for example.

[0068] Figure 4 The illustration shows an example of a target work area obtained by work area acquisition unit 302. Figure 4 In the diagram, areas 401 and 402 indicate aluminum framing, which is a building material used for window frames. The work to be performed here is for workers to wipe the aluminum framing as it moves along a conveyor belt using work cloths.

[0069] Furthermore, depending on the acquired images, it's possible that feature points forming the target area may not be accurately obtained. For example, in some cases, the worker's body may partially obscure the image.

[0070] If the number of feature points obtained this time is equal to or greater than a certain number, and the difference between the number of feature points of the obtained shape and the number of feature points of the previously obtained target region is equal to or less than a certain value, the previously obtained target region can be used as a substitute. The determination of the difference between the shape obtained from the obtained feature points and the previously obtained target region can be performed based on whether the difference in the Euclidean distances of the coordinate points is equal to or less than a threshold. The Euclidean distances of the coordinate points can be calculated in association with each feature point.

[0071] The object to be detected by the object detection unit 303 can be, for example, a worker's hand, or equipment used for work. When performing work, the worker's hand or equipment overlaps with the work area, thus the location where the object is detected is determined as the work completion area. Therefore, even when a colorless protective agent is applied to the component or the component is wiped with alcohol, the area including the location where the object is detected is determined as the work completion area, thereby allowing for an appropriate work ratio to be obtained.

[0072] When detecting an object, a template image can be used as a template for matching the target object. Furthermore, deep learning can be used to detect target objects.

[0073] For example, Figure 5 The illustration shows an example of detecting objects using a model pre-trained on a worksheet. The detected area of ​​the worksheet is region 501.

[0074] The region generated by the region correction unit 304 is calculated using the target working region obtained by the working region acquisition unit 302, the completed working region obtained by the object detection unit 303, and a preset correction standard. The preset correction standard can be a shape calculated from a model image including a pre-input target working region. Alternatively, the correction standard can be the target working region initially obtained by performing this process. The correction standard can also be a shape fixed and maintained internally by the system.

[0075] Now will be used Figure 6 Describe an example of calculating the correction standard from a model image.

[0076] Figure 6 The illustration shows an image 600 captured from above the target working region. The target working region of image 600 is designated as region 601. With region 601 serving as the calibration standard, a grid for the frequency map is generated based on the calibration standard.

[0077] The frequency map grid can be generated by dividing the regions corresponding to the calibration criteria into bounding boxes in the vertical and horizontal directions. The units used for segmentation can be specified by the user or determined based on the physical dimensions of the target region and the object being detected. For example, the intention is to generate a heat map of region 601 with lower bounds of 10 cm and 1 m. In this case, if the length of the lower bound in the image of the target region is 500 pixels, then the block size in the horizontal direction can be set to 50 pixels. The same applies to the vertical direction to calculate the block size.

[0078] In other words, to calculate the block size, the edges of the region forming in the actual object of the working target are correlated with and the edges of the working target region forming the captured image are compared in length. The unit used for segmentation can be the smallest pixel in the captured image as a reference.

[0079] The calibration standard can be stored in the storage device 109 and read from the storage device 109 during the processing of the region calibration unit 304 or loaded into the memory of the recording device 100 before the processing of the region calibration unit 304.

[0080] In the area correction unit 304, the shape of the target area and the completed area within the target area is corrected to match the correction standard.

[0081] When performing shape corrections on region 601, which serves as the correction standard generated as described above, region 401, which serves as the target area for work, and region 501, which serves as the completed area for work, regions 401 and 501 are scaled to match the dimensions of region 601 in the vertical and horizontal directions.

[0082] The frequency map generation unit 305 applies the mesh obtained when generating region 601 to the corrected target area and the corrected completed area obtained by scaling, and determines that the mesh that overlaps with the completed area is detected.

[0083] Whether to include completed work areas can be determined based on whether the ratio of completed work areas relative to each grid is greater than a threshold. When each grid is determined to be completed, the number of detections in the frequency map is counted upwards.

[0084] The results obtained from the counting are stored as numerical data for each grid. For example, using the top left of the grid as a reference and assigning numbers such as grid 1, 2, etc., the number of detections for each grid can be associated and stored.

[0085] A series of frequency maps of work status, converted into numerical values, can be generated from multiple work target areas and work completion areas obtained through the processing of the repeated image acquisition unit 301, the work area acquisition unit 302, and the object detection unit 303.

[0086] The heatmap generation unit 306 generates a heatmap using a color scale from the frequency map information obtained by the frequency map generation unit 305. At this time, the heatmap is displayed in the shape of the frequency map obtained by the working area acquisition unit 302, which has been scaled to match the working area.

[0087] Furthermore, if the feature points that form the target area cannot be obtained correctly and the previously obtained target area is used as a substitute, the shape is scaled to match the previously obtained target area.

[0088] Figure 7 The illustration shows an image 700 of a heatmap superimposed on the target area and a grid 701 of the heatmap.

[0089] Grid 701 is located at the edge of the target working area, therefore a portion of the grid is not included in the working area. In such cases, only the portion included in the working area is displayed as the grid in the heatmap.

[0090] The following will refer to Figure 8Detailed description of the flowchart shown in the figure Figure 3 Each function of the recording device 100 illustrated in the figure. Figure 8 The processing begins when the user starts the information processing system 200. Note that S indicates the step number.

[0091] In S8001, the image acquisition unit 301 acquires an image. The image acquired here is then used to perform the processing in S8002.

[0092] In S8002, the object detection unit 303 detects objects used to identify and match the work completion area of ​​the currently performed task. Figure 5 In the example shown in the image, this is a cloth used to wipe the aluminum frame.

[0093] In S8003, the work area acquisition unit 302 detects and matches the work target area of ​​the currently executing task. Figure 4 In the example shown in the image, this is an aluminum frame.

[0094] It should be noted that, Figure 8 In the process, S8002 and S8003 are executed sequentially, but the detection order can be reversed.

[0095] In S8004, CPU 101 determines whether both the working area and the object have been detected in the processing of S8002 and S8003. If at least one has not been detected ("No" in S8004), then the processing returns to S8001, the image is reacquired, and the processing after S8001 is executed.

[0096] If both have been detected ("yes" in S8004), then CPU 101 executes the processing of S8005.

[0097] In S8005, the area correction unit 304 performs correction processing. Here, the shape correction of the area is performed using the object area and working area obtained in S8002 and S8003, as well as the correction standard.

[0098] In S8006, the frequency diagram generation unit 305 uses the region generated in S8005 to generate a frequency diagram. By repeating the processes of S8001 to S8005 in this flowchart, a frequency diagram indicating the status of work being performed on the work area can be generated.

[0099] In the processing of S8007, for the frequency map generated in S8006, inverse shape correction from the calibration standard to the working area is performed to generate a heat map.

[0100] In S8008, the overlay image display unit 307 overlays a heat map onto the image. The heat map generated in S8007 is overlaid on the working area obtained by performing the processing of S8003 on the image obtained in S8001. The overlay result is presented to the user via the display provided in the display unit 108.

[0101] By repeating the processes from S8001 to S8008, the ratio of work to work area in matching work conditions can be visualized.

[0102] Despite Figure 8 Not illustrated, but for example, the determination to terminate the S8008 process can be made based on whether a certain value is exceeded for a certain area or a larger heatmap of the target area. The determination can also be made if the target area detection process and object detection process have not been performed for a certain period of time. Alternatively, the determination can be made if a certain period of time has elapsed since the initial object detection time.

[0103] In the first embodiment, when generating the frequency map, the target area and the completed area of ​​the work are calibrated based on a calibration standard. However, in cases such as belt conveyors where the movement distance of the parts is short and the size of the parts to be detected from the image does not change even if the parts are moved, the target area and the completed area of ​​the work can be used with uncalibrated shapes.

[0104] In this scenario, a frequency map is generated based on the target work area and the completed work area obtained by the work area acquisition unit 302 and the object detection unit 303. Furthermore, a heatmap is generated by adjusting the position of the generated frequency map to match the position of the target work area in the image obtained by the image acquisition unit 301. Additionally, the generated heatmap is displayed overlaid based on the position of the target work area in the image. Therefore, even when the size of the part to be detected from the image remains unchanged but the position of the part changes, the work ratio for the target work area can be appropriately visualized for the user.

[0105] Second Embodiment

[0106] In the method according to the first embodiment described above, the frequency map is corrected by scaling the working target area in the vertical and horizontal directions, and a heat map is displayed to match the working target area.

[0107] In the second embodiment, shape correction will be described in cases where the shape of the target area is complex and there is a tilt angle between the camera and the target area.

[0108] In this embodiment, the processing of the region correction unit 304, the frequency map generation unit 305, and the heat map generation unit 306 will be described. The image acquisition unit, the working area acquisition unit 302, the object detection unit 303, and the overlay image display unit 307 are as described in the first embodiment and therefore will not be described.

[0109] The correction criteria pre-stored in the region correction unit 304 are as described in the first embodiment in terms of content.

[0110] In the region correction unit 304, the working target region, including the object detection region of the object detection unit 303, obtained by the working region acquisition unit 302, is normalized to the size of a certain standard region (correction standard). An example of correction using projective transformation will now be used. Figure 9 , Figure 10 , Figure 11 , Figure 12 and Figure 13 Describe how normalization is performed.

[0111] Figure 9 The illustration shows a captured image 900 obtained by camera 110. The captured image 900 includes a region 901, which is a target area obtained by the working area acquisition unit 302. Region 901 is composed of coordinate points 902, 903, 904, and 905.

[0112] Furthermore, the captured image 900 includes region 911, which is the completed work area detected by object detection unit 303. Region 911 is composed of coordinate points 912, 913, 914, and 915. Region correction unit 304 stores preset correction standards.

[0113] Calibration standards in Figure 10 The diagram is shown in the image. Figure 10 The diagram illustrates region 1000 corresponding to the calibration standard, and the coordinate points 1001, 1002, 1003, and 1004 constituting region 1000.

[0114] In this embodiment, region 1000 is a region of a defined size maintained within the system, and it can be stored in storage device 109 and read out by recording device 100 when performing the processing of region correction unit 304.

[0115] The transformation matrix for projective transformation is calculated from the target working region and the calibration standard. Coordinates 902 and 1001, 903 and 1002, 904 and 1003, and 905 and 1004 are correlated and a simultaneous equation is generated, from which the transformation matrix is ​​calculated.

[0116] By applying the transformation matrix obtained here to the work completion area, a corrected work completion area matching the correction standard (area 1000) can be obtained. The corrected work completion area can be as follows: Figure 11 As shown in the image.

[0117] Figure 11 The illustration shows region 1110 within region 1000, which serves as the rectified work completion area. Furthermore, region 1110 is composed of coordinate points 1111, 1112, 1113, and 1114. Coordinate points 1111, 1112, 1113, and 1114 are associated with coordinate points 912, 913, 914, and 915 of the captured image 900.

[0118] Furthermore, in this embodiment, a method for performing a projective transformation on a working target region consisting of four points has been described. However, the coordinate points constituting the working target region can be three points or five or more. In such a case, the bounding box of the working target region, the coordinate points constituting the bounding box, and the coordinate points constituting the bounding box of the correction criterion are obtained, and a projective transformation based on the obtained coordinate points is performed to obtain a transformation matrix.

[0119] The frequency map generated by the frequency map generation unit 305 is generated using a region corresponding to the calibration standard as a reference. Now, we will use... Figure 12 The example of a frequency map shown in the figure describes a method for generating frequency maps. Figure 12 The diagram illustrates a grid 1201 that indicates the grid forming the frequency map and a frequency region 1202 that indicates frequency information.

[0120] Region 1000 is divided into predetermined blocks by grid 1201. It is determined whether a completed work area is included in each grid 1201. This can be determined based on whether the ratio of completed work areas relative to each grid is greater than a threshold. When each grid is determined to be completed, the number of detections in the frequency map is counted upwards.

[0121] The captured images are repeatedly acquired, and for each captured image, a corrected target region and a completed region are generated to produce a frequency map. The region obtained as a result of the counting corresponds, for example, to region 1202.

[0122] According to the embodiment described herein, the results obtained from the counting are illustrated as region 1202. However, the results can be stored as numerical data for each grid by the frequency map generation unit 305.

[0123] For example, the top left of the grid can be used as a reference and assigned numbers such as grid 1, 2, etc. The number of detections for each grid can be associated and stored.

[0124] The heatmap generation unit 306 generates a heatmap by performing shape correction on the frequency map generated by the frequency map generation unit 305 to match the working target region in the image obtained by the image acquisition unit 301.

[0125] The frequency diagrams illustrated in regions 1000 and 1202 will now be described to match... Figure 9 Example of working target area 901 shown in the figure.

[0126] The transformation from region 1000 to region 901 can be performed using the inverse of the transformation matrix obtained by region correction unit 304.

[0127] A transformed heatmap is generated from the frequency map information of the target working area obtained by matching and transforming region 1202 of each grid of the frequency map for display.

[0128] To generate a frequency map for display of matching region 901, the shape of each block needs to be scaled according to the shape of the target area. For example, in the case of a shape such as region 901, the grid size of region 1202 needs to be reduced for the upper part of the shape, and the grid size needs to be increased for the lower part of the shape.

[0129] Heatmap information can be generated with a scaled grid size, or the minimum size of the grid used for display can be changed according to the shape of the working area of ​​the heatmap display target.

[0130] For example, the minimum grid size for the frequency map used in the heatmap can be determined to match the minimum size of the region of each grid obtained when the region 1202 of each block of the frequency map is transformed using the inverse matrix of the transformation matrix obtained by the region correction unit 304. Furthermore, the frequency information can be transformed to match the grid generated to match the target working region.

[0131] Now based on Figure 13 The grid 1300 and frequency information 1301 and 1302 shown in the figure describe a frequency information conversion method. The grid 1300 is part of a grid generated to match the target working region.

[0132] Frequency information 1301 and 1302 are part of a grid of a frequency map obtained by transformation to match the target working area obtained by transformation from region 1000 to region 901.

[0133] When the frequency information differs between frequency information 1301 and 1302, the value of grid 1300 can be determined to match the ratio of the regions relative to the grid. When the region ratio of frequency information 1301 and 1302 is a:b, the value of frequency information 1301 is n, and the value of frequency information 1302 is m, the value obtained via (a / a+b)×m+(b / a+b)×n can be calculated as the frequency information of grid 1300. Alternatively, the value of the largest region among the regions of frequency information 1301 and 1302 can be used.

[0134] exist Figure 14 The figure illustrates an example of a heatmap display generated according to the second embodiment. Color stops 1401, a target area 901, and a heatmap area 1403 are displayed for the captured image 1400.

[0135] Color scales can be represented by hue or by the brightness of a color. Figure 14 The diagram shows 5 levels, but the number of levels is not limited.

[0136] Furthermore, in the second embodiment, projective transformation is used as the normalization method, but the normalization is not limited to projective transformation alone. For example, free form deformation (FFD), affine transformation, etc., can be used.

[0137] Third Embodiment

[0138] In the first and second embodiments, the heatmap is superimposed on the working target area of ​​the captured image, but the heatmap may, for example, be superimposed on a pre-obtained model image.

[0139] At this time, the heatmap generation unit 306 deforms the frequency map obtained by the frequency map generation unit 305 to match the working target region of the model image. The deformation method can be similar to the method described in the first and second embodiments.

[0140] Furthermore, the display method used for heatmap display can vary depending on the image on which the heatmap is to be overlaid. For example, if the size of the target area is equal to or smaller than a threshold, the transparency of the heatmap's color bars can be reduced, and if the size of the target area exceeds the threshold, the transparency of the heatmap's color bars can be increased. Moreover, the thickness of the bounding box surrounding the target area can be varied depending on its size. When the target area is small, the bounding box can be thin, and when the target area is large, the bounding box can be thick, thus adjusting the visibility of the heatmap display within the target area.

[0141] Fourth embodiment

[0142] In the fourth embodiment, a mechanism for notifying users based on the obtained frequency maps and heatmaps will be described.

[0143] like Figure 15 As shown, towards Figure 3 The recording device 100 provides the function of a determination unit 1508. The determination unit 1508 pre-obtains threshold information corresponding to a standard used to notify the user. The threshold information is a numerical value representing the percentage of regions in the frequency map where the number of detections obtained from the frequency map exceeds a certain value. For example, the threshold could be that regions with more than three detections represent 70% or more of the frequency map.

[0144] If a threshold is exceeded, the user is notified that the task is complete. The notification method can be a warning light (not shown) or text or an icon indicating that the task is complete can be displayed on the display unit 108.

[0145] Furthermore, if the threshold is not exceeded and the target area and completed area have not been detected for a certain period of time, the user can be notified.

[0146] Furthermore, the threshold can be the numerical value of the difference between the maximum and minimum number of detections in the frequency graph.

[0147] By providing a threshold for the work rate in this way and providing a notification method when the threshold is exceeded or not exceeded, in addition to visualizing the work status, it can also show users (workers) whether the work has been performed correctly or not.

[0148] Furthermore, by defining the threshold as the numerical difference between the maximum and minimum number of detections in the frequency graph, unevenness in the work can be shown to the user (staff). It should be noted that, regardless of the foregoing, the notification method and threshold setting method can be appropriately modified.

[0149] Other embodiments

[0150] Embodiments of this disclosure can also be implemented by a computer that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a 'non-transitory computer-readable storage medium') to perform one or more functions of the above embodiments and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing one or more functions of the above embodiments, and by a method performed by a computer of the system or device by, for example, reading and executing computer-executable instructions from the storage medium to perform one or more functions of the above embodiments and / or controlling one or more circuits to perform one or more functions of the above embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include separate computers or networks of separate processors to read and execute computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), storage devices for distributed computing systems, optical discs (such as CDs, DVDs, or Blu-ray discs). TM One or more of the following: flash memory devices, memory cards, etc.

[0151] Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0152] While this disclosure has been described with reference to exemplary embodiments, it is to be understood that this disclosure is not limited to the disclosed exemplary embodiments. The scope of the appended claims is to be given the broadest interpretation in order to cover all such modifications and equivalent structures and functions.

Claims

1. An information processing apparatus, comprising: The acquisition unit acquires images of the captured working area and the object; A first detection unit detects a working area from the image; A second detection unit detects object regions from the image; A frequency map generation unit generates a frequency map based on the number of detections of object regions in each grid of the working area; A heat map generation unit generates a heat map when the frequency map is made to correspond to the working area detected by the first detection unit. as well as The display control unit overlays a heat map on a display device at a location based on the working area.

2. The information processing apparatus according to claim 1, further comprising: A correction unit that corrects the shape of the working area and the object area.

3. The information processing apparatus according to claim 2, wherein... The frequency map generation unit generates a frequency map based on the number of object regions detected in each grid within the working area corrected by the correction unit.

4. The information processing apparatus according to claim 2, wherein... The correction unit corrects the shape of the working area and the object area by enlarging or reducing the shape of the working area and the object area in the vertical or horizontal direction to match the standard area used for correction.

5. The information processing apparatus according to claim 2, wherein... The correction unit corrects the shapes of the working area and the object area by normalizing the shapes of the working area and the object area to match the standard area used for correction.

6. The information processing apparatus according to claim 5, wherein The correction unit normalizes the working area and the object area to match the standard area, and the heatmap generation unit converts the frequency map to correspond to the working area of ​​the image obtained by the acquisition unit from the standard area.

7. The information processing apparatus according to claim 5, wherein... The correction unit performs the normalization via projective transformation.

8. The information processing apparatus according to claim 5, wherein The correction unit performs the normalization via free-form deformation.

9. The information processing apparatus according to claim 5, wherein The correction unit performs the normalization via an affine transformation.

10. The information processing apparatus according to claim 5, wherein The standard area is determined based on a pre-captured work area.

11. The information processing apparatus according to claim 1, wherein If, compared to the working area detected in the previous detection, there are undetected coordinate points in the working area detected in the current detection, and the difference between the number of coordinate points detected in the current detection and the number of coordinate points detected in the previous detection is equal to or less than a threshold, the first detection unit uses the working area detected in the previous detection as the working area detected in the current detection.

12. The information processing apparatus according to claim 1, wherein The heatmap generation unit updates the grid size of the heatmap based on the shape of the working area detected by the first detection unit.

13. The information processing apparatus according to claim 1, wherein The heat map generation unit updates at least one of the display color and line thickness based on the shape of the working area detected by the first detection unit.

14. The information processing apparatus according to claim 1, further comprising: The warning unit issues a warning if the frequency and number of each grid in the frequency map are not greater than a threshold and no working area or object area has been detected for a certain period of time.

15. A method for controlling an information device, the method comprising: Obtain images of the captured work area and objects; Perform a first detection from the image detection working area; Perform a second detection of the object region from the image; A frequency map is generated based on the number of times the object region is detected in each grid of the working area; A heatmap is generated when the frequency map corresponds to the working area detected by the first detection. as well as Heatmaps are overlaid on a display device at locations based on the working area.

16. A program product for causing a computer to perform the method of the control information processing apparatus according to claim 15.

17. A non-transitory computer-readable storage medium storing a program for causing a computer to perform the method of the control information processing apparatus according to claim 15.

Images