Information processing device and its control method, program, and storage medium

Abstract

The present invention provides an information processing device that can appropriately visualize the proportion of work performed within a given work area. [Solution] The system includes an acquisition unit that acquires images of a work area and an object, a first detection unit that detects the work area from the image, a second detection unit that detects the object area from the image, a frequency map creation unit that creates a frequency map based on the number of times the object area is detected for each grid in the work area, a heat map creation unit that creates a heat map by corresponding the frequency map to the work area detected by the first detection unit, and a display control unit that overlays the heat map onto the work area and displays it on a display unit.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus for visualizing work ratios.

Background Art

[0002] In a production site, it is confirmed whether work standards are met using an image processing sensor.

[0003] For example, Patent Document 1 discloses a work management system for managing an operation including a repetitive operation in which an operator repeatedly moves a polishing tool. The positions of working elements such as the operator's fingers and the polishing tool are specified, the movement vectors of the working elements are stored, and the quality of the work is managed.

[0004] Also, depending on the work standards, there may be a case where it is desired to visualize the ratio of work done. For example, in the case of painting with a paint or the like, the work ratio can be visualized using color information. Also, even when color information cannot be used, work management using an image processing sensor can be performed by the technique disclosed in Patent Document 2. Patent Document 2 discloses a technique for creating a heat map for visualizing the residence information of a moving body from the position information of the moving body and the residence information for each predetermined measurement period.

[0005] From the above, when there is no relative position change between the image processing sensor and the article, or when there are changes in color or shape, work management using the image processing sensor is possible.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] However, within the scope of the prior art disclosed in the aforementioned patent documents, when applying a colorless protective agent or wiping with alcohol to parts moving on a conveyor belt, it is not possible to determine the proportion of work performed by using the change in the apparent work area.

[0008] The inability to properly obtain data on the proportion of work performed makes it difficult to determine whether work is being done evenly and without bias across the work area.

[0009] The present invention has been made in view of the above-mentioned problems, and its objective is to provide an information processing device that can appropriately visualize the work ratio relative to the work area. [Means for solving the problem]

[0010] The information processing device according to the present invention is The relative position to the imaging device changes. Work area Area Image taken multiple An acquisition means for acquiring an image, and the multiple A first detection means for detecting a work area from an image, and the When multiple images contain objects, the multiple A second detection means for detecting object regions from an image, Correction means for correcting the shape of the work area when the size of the work area on the images differs among the plurality of images, The aforementioned Corrected by the correction means The system is characterized by comprising: frequency map creation means for creating a frequency map based on the number of times the object region is detected for each grid in the work area; heat map creation means for creating a heat map by corresponding the frequency map to the work area detected by the first detection means; and display control means for displaying the heat map superimposed on a display means based on the work area. [Effects of the Invention]

[0011] According to the present invention, it becomes possible to appropriately visualize the proportion of work to the work area. [Brief explanation of the drawing]

[0012] [Figure 1]Block diagram showing an example of the hardware configuration of the recording device in the first embodiment. [Figure 2] Diagram showing a schematic configuration example of an information processing system including a recording device. [Figure 3] Block diagram showing a functional configuration example of the recording device. [Figure 4] Diagram showing an example of the work target area acquired by the work area acquisition unit. [Figure 5] Diagram for explaining the processing of the object detection unit. [Figure 6] Diagram for explaining the processing of the area correction unit. [Figure 7] Diagram for explaining the processing of the heat map creation unit. [Figure 8] Flowchart showing the procedure for creating a heat map. [Figure 9] Diagram for explaining the processing of the area correction unit in the second embodiment. [Figure 10] Diagram for explaining the correction standard of the area correction unit. [Figure 11] Diagram showing the worked area after correction. [Figure 12] Diagram for explaining the processing of the frequency map creation unit. [Figure 13] Diagram for explaining the processing of correcting the frequency information of the frequency map. [Figure 14] Diagram showing an example of heat map display. [Figure 15] Block diagram showing a functional configuration example of the recording device in the third embodiment.

Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant explanations are omitted. (First Embodiment) In the first embodiment, an information processing device is described that corrects a frequency map showing the worked areas where work has been performed on the work area acquired from the captured image, according to the shape of the work area, and overlays it as a heat map.

[0014] In this embodiment, a process for correcting the heatmap display using the vertical and horizontal dimensions of the work area will be described.

[0015] Figure 1 is a block diagram showing an example of the hardware configuration of the recording device 100 in the first embodiment.

[0016] The recording device 100 includes a CPU 101, ROM 102, RAM 103, storage 104, and communication I / F 105. The CPU 101, ROM 102, RAM 103, storage 104, and communication I / F 105 are connected to an internal bus 106.

[0017] The CPU 101 is a central processing unit that comprehensively controls the operation of the recording device 100. The ROM 102 is a non-volatile memory that stores programs and various initial setting data necessary for the CPU 101 to execute processing. The RAM 103 is a volatile memory that functions as the main memory and work area of ​​the CPU 101, and temporarily stores programs, image data, etc. When executing processing, the CPU 101 loads the necessary programs from the ROM 102 into the RAM 103 and executes those programs to realize various functional operations.

[0018] Storage 104 is a storage device with a larger capacity compared to RAM 103, and for example, an HDD (hard disk drive) or SSD (solid state drive) is used. Storage 104 stores the OS (operating system software) executed by CPU 101. It also stores various programs for acquiring the work target area, acquiring the worked area, area correction, frequency map creation, and heat map creation in this embodiment, as well as various parameters and data related to the programs. Furthermore, storage 104 can also record image data acquired via the network.

[0019] When the CPU 101 is powered on or otherwise started, it executes a boot program stored in the ROM 102. This boot program reads the OS stored in the storage 104 and loads it into the RAM 103.

[0020] After the OS starts, the user can, for example, via the operation unit 107 shown in Figure 2, issue instructions to start the work target area acquisition program, the worked area acquisition program, the area correction program, the frequency map creation program, and the heat map creation program. The CPU 101 then reads each program from the storage 104 and loads it into the RAM 103. As a result, the CPU 101 creates a frequency map of the worked area from the corrected worked area. Various data used for the operation of the program to superimpose the heat map on the work target area of ​​the captured image are also stored in and read from the RAM 103.

[0021] Communication I / F105 is, for example, a LAN (Local Area Network) interface, and communicates with network cameras and other computers via the network.

[0022] In this embodiment, the images handled by each program, the processed area acquisition program, the area correction program, the frequency map creation program, the heat map creation program, and the various data used in each program may be acquired via a network.

[0023] The recording device 100 can be configured as a single device having the configuration shown in Figure 1, such as a personal computer (PC) or tablet PC. However, the configuration shown in Figure 1 may be made up of separate hardware. In other words, the recording device 100 may be made up of multiple devices. Furthermore, calculation processing associated with various analyses and judgments may be processed using a GPU (not shown).

[0024] Figure 2 is a diagram showing a schematic configuration example of an information processing system 200 equipped with a recording device 100 in this embodiment.

[0025] The information processing system 200 is comprised of a recording device 100, a storage device 109, and a camera 110.

[0026] The recording device 100, the storage device 109, and the camera 110 are each connected to each other via a network 111 so that they can communicate with one another.

[0027] Network 111 can be, for example, a LAN. The communication standard, size, and configuration of network 111 are not restricted, as long as it enables communication between the recording device 100, camera 110, and storage device 109. Furthermore, the physical connection to network 111 may be wired or wireless. Additionally, the connection between the recording device 100, storage device 109, and camera 110 is not limited to network 111; it may also be connected via, for example, USB.

[0028] In addition to the configuration shown in Figure 1, the recording device 100 may also include an operation unit 107 and a display unit 108 as part of its hardware configuration. Here, the operation unit 107 includes a pointing device such as a keyboard or mouse. The display unit 108 includes a monitor such as a liquid crystal display (LCD) and is a display device for viewing information when the user (operator) operates the recording device 100.

[0029] Camera 110 is, for example, a network camera, and is an imaging device that captures an image within its imaging range and transmits the captured image to the recording device 100 via the network 111. Camera 110 may also be a PTZ camera (Pan Tilt Zoom camera) configured to change its imaging area. In this case, camera 110 may have the function of transmitting imaging parameter information, including imaging direction and imaging angle of view such as pan angle, tilt angle, and zoom magnification, to the recording device 100 via the network 111.

[0030] Furthermore, this embodiment describes a case where camera 110 is a network camera used as a surveillance camera, but it is not limited to this. For example, camera 110 may be a digital still camera, a digital video camera, a smartphone or tablet terminal equipped with a camera function, an industrial camera, an in-vehicle camera, a wearable camera, etc. The recording device 100 receives the captured image taken by camera 110 as an input image from camera 110 via network 111, and performs the processing described later on the received input image.

[0031] The camera 110 may transmit captured images to the storage device 109 or other storage devices of other devices via the network 111. In this case, the recording device 100 may receive the captured images stored in the storage device 109 or other storage devices of other devices as input images via the network 111. Here, the input image is not limited to the captured image taken by the camera 110. For example, the input image may be a partial image, which is a part of the captured image. Also, the configuration of the information processing system 200 shown in Figure 2 is just an example, and each device may be integrated, such as a tablet PC. In other words, the recording device 100 may also have the functions of the camera 110.

[0032] The recording device 100 acquires the work area and the completed work area from the image acquired via the camera 110. A frequency map is created from both acquired areas, and the heat map display is corrected according to the frequency map and the shape of the work area and superimposed on the image.

[0033] The work area refers to the working range of the object being worked on, or its parts. The completed work area refers to the area where the worker's hands or work tools were detected.

[0034] In generating the frequency map, the number of operations is counted for any area within the work area. The number of operations is calculated based on the number of times the worker's hand or work tools are detected. The created heat map is displayed via the display unit 108 on a monitor or other display device, allowing the user to view it.

[0035] Figure 3 is a block diagram showing an example of the functional configuration of the recording device 100 in this embodiment.

[0036] As shown in Figure 3, the recording device 100 includes an image acquisition unit 301, a work area acquisition unit 302, an object detection unit 303, a region correction unit 304, a frequency map creation unit 305, a heat map creation unit 306, and a superimposed image display unit 307. Furthermore, at least some of the functions of each element of the recording device 100 shown in Figure 3 can be realized by the CPU 101 executing a program. However, at least some of the elements of the recording device 100 shown in Figure 3 may be configured to operate as dedicated hardware. In this case, the dedicated hardware operates based on the control of the CPU 101.

[0037] The functions of the recording device 100 will be explained below using Figure 3.

[0038] The image acquisition unit 301 receives images acquired via the camera 110 or the like.

[0039] The work area acquisition unit 302 detects the work area from the image acquired by the image acquisition unit 301.

[0040] The object detection unit 303 detects arbitrary objects from the image acquired by the image acquisition unit 301. The objects detected here include the hands of a person performing a task, tools used in the task, equipment, etc.

[0041] The area correction unit 304 transforms the shape of the work target area and the completed work area acquired by the work area acquisition unit 302 and the object detection unit 303 based on correction criteria stored in the area correction unit 304 that have been set in advance.

[0042] The frequency map creation unit 305 creates a frequency map based on the work area and completed area, whose shapes have been corrected based on the correction criteria. For example, if the work area is a part moving on a conveyor belt, and images are acquired from the image acquisition unit 301 of the recording device 100 located in a fixed position, the size of the part detected in an image acquired at one point in time may differ from the size of the part detected in an image acquired at another point in time. The same applies when the completed area is a region including the hands of a person working on a conveyor belt, and images are acquired from the image acquisition unit 301 of the recording device 100 located in a fixed position. Even in such cases, by creating a frequency map based on the work area and completed area, which have been corrected for differences in the size of the region including the parts and hands, the work ratio can be appropriately obtained.

[0043] The heatmap creation unit 306 creates a heatmap by correcting the shape of the frequency map created by the frequency map creation unit 305 to match the work area in the image acquired by the image acquisition unit 301. In this way, the heatmap is created by correcting the shape of the frequency map to match the work area in the image acquired by the image acquisition unit 301, rather than the work area whose shape has been corrected based on the correction criteria. This makes it possible to create a heatmap for the work area that the user is viewing in real time. The superimposed image display unit 307 superimposes and displays the heatmap created by the heatmap creation unit 306 onto the image at a position based on the work area. For example, if the work area is a part moving on a conveyor belt, the created heatmap is superimposed on the image in accordance with the movement of the work area. In other words, the created heatmap moves and is displayed in accordance with the movement of the work area. This allows the user to appropriately obtain the work ratio relative to the work area, even when the work area is moving.

[0044] The functions of the recording device 100, specifically the work area acquisition unit 302, object detection unit 303, area correction unit 304, frequency map creation unit 305, and heat map creation unit 306, will be explained in detail below using Figures 4, 5, 6, and 7.

[0045] The work area acquired by the work area acquisition unit 302 is, for example, a part to be worked on or an area indicating the scope of work to be performed. Feature points are extracted from the part or the area indicating the scope of work to be performed, and the area enclosed by these feature points is defined as the work area. Feature points may be extracted using, for example, SIFT (scale-invariant feature transformation) to extract the corners of the target part or area. Alternatively, other methods may be used for feature point extraction, and the type of method is not limited.

[0046] Alternatively, markers such as 2D barcodes may be used to acquire the work area. In this case, markers are attached to the parts or areas to be worked on in advance, and the coordinate points indicating the boundaries of the area are obtained by detecting the markers. The method for detecting the area is not limited to the method described above; for example, methods such as detecting the area using deep learning may also be used.

[0047] Figure 4 shows an example of a work area acquired by the work area acquisition unit 302. In Figure 4, areas 401 and 402 represent aluminum sashes, which are building materials used as window frames. The scenario assumes a worker wiping aluminum sashes moving on a conveyor belt with a work cloth.

[0048] Furthermore, depending on the image obtained, it may not be possible to correctly acquire the feature points that make up the work area. For example, this may occur if part of the area is hidden by the worker's body.

[0049] If the number of feature points acquired this time exceeds a certain number, and the difference between the number of feature points of the acquired shape and the number of feature points of the previously acquired work area is less than or equal to a certain value, the previously acquired work area may be used as a substitute. The difference between the shape obtained from the acquired feature points and the previously acquired work area may be determined by whether the difference in the Euclidean distance of the coordinate points is less than or equal to a threshold. The Euclidean distance of the coordinate points may be calculated in association with each feature point.

[0050] The object detection unit 303 can detect any object, such as a worker's hand or work equipment. When performing work, the work area overlaps with the worker's hand or equipment, so the location where an object is detected can be considered part of the completed work area. This allows for the accurate acquisition of work progress even when applying a colorless protective agent to a part or wiping a part with alcohol, by considering the area including the location where an object was detected as part of the completed work area.

[0051] For arbitrary object detection, template matching may be used, where the target object is used as a template image. Alternatively, deep learning may be used to detect the target object.

[0052] For example, Figure 5 shows an example of object detection using a model that has been previously trained on a work cloth. The region where the work cloth was detected is region 501.

[0053] The region created by the region correction unit 304 is calculated using the work target region acquired by the work region acquisition unit 302, the worked region acquired by the object detection unit 303, and a predetermined correction criterion. The predetermined correction criterion may be a shape calculated from a model image that includes a pre-input work target region. Alternatively, it may be the work target region acquired first when this process is executed. Alternatively, it may be a rectangle that is fixed within the system.

[0054] Figure 6 illustrates an example of calculating correction criteria from a model image.

[0055] Figure 6 shows image 600, which is a top-down view of the work area. The work area in image 600 is shown as area 601. When area 601 is used as the correction criterion, a grid for frequency mapping is created for the correction criterion.

[0056] A grid for frequency maps may be created by dividing the bounding rectangle of the correction reference area vertically and horizontally. The division units can be arbitrarily specified by the user or determined based on the physical size of the work area and the detected objects. For example, suppose the bottom edge of area 601 is 1m and we want to create a heatmap in 10cm units. In that case, if the length of the bottom edge of the work area on the image is 500 pixels, the horizontal block size should be 50 pixels. The vertical block size can be calculated similarly.

[0057] In other words, the block size may be calculated by comparing the lengths of the edges that constitute a region in the actual object being worked on with the edges that constitute the working region in the captured image. The unit of division may be based on the smallest pixel in the captured image.

[0058] The correction criteria are stored in the storage device 109 and may be read from the storage device 109 when the area correction unit 304 processes, or they may be loaded into the memory on the recording device 100 before the area correction unit 304 processes.

[0059] The area correction unit 304 corrects the shape of the work area and the work area located within the work area according to the correction criteria.

[0060] As described above, when performing shape correction on the correction standard area 601, the work target area 401, and the completed work area 501, areas 401 and 501 are scaled to match the vertical and horizontal size of area 601.

[0061] The frequency map creation unit 305 applies the grid obtained when creating region 601 to the corrected work target region and the corrected work area obtained by scaling, and determines that detection has been performed for grids that overlap with the work area.

[0062] Whether a grid contains a completed area can be determined by whether the proportion of the completed area within each grid exceeds a threshold. When each grid is determined to be completed, it is counted as a detection in the frequency map.

[0063] The results obtained from the count are saved as numerical data for each grid. For example, you could number the grids 1, 2, ... starting from the top left corner and save the number of detections for each grid in association with that number.

[0064] By repeatedly performing the processes of the image acquisition unit 301, the work area acquisition unit 302, and the object detection unit 303, a frequency map can be created that quantifies a series of work statuses from multiple work target areas and completed areas acquired.

[0065] The heatmap creation unit 306 generates a heatmap using a color scale from the frequency map information obtained by the frequency map creation unit 305. At this time, the heatmap is displayed by scaling the shape of the frequency map to match the working area acquisition unit 302.

[0066] Furthermore, if the feature points constituting the work area were not correctly acquired and the previously acquired work area was used as a substitute, the shape will be scaled to match the previously acquired work area.

[0067] Figure 7 shows image 700 with a heatmap superimposed on the work area, and the heatmap grid 701.

[0068] Grid 701 is located at the edge of the work area, so part of the grid is not included in the work area. In such cases, only the portion included in the work area will be displayed as the grid in the heatmap.

[0069] The following explanation details each function of the recording device 100 shown in Figure 3, based on the flowchart shown in Figure 8. The process in Figure 8 begins when the user starts the information processing system 200. Note that "S" indicates the step number.

[0070] In S8001, the image acquisition unit 301 acquires an image. The image acquired here is then used to execute the process in S8002.

[0071] In S8002, the object detection unit 303 detects an object to identify a completed area in accordance with the work being performed. In the example shown in Figure 5, it is a cloth used to wipe aluminum window frames.

[0072] In S8003, the work area detection unit 302 detects the work area corresponding to the work being performed. In the example shown in Figure 4, this is an aluminum window frame.

[0073] Note that in Figure 8, steps S8002 and S8003 were executed in that order, but this detection order can be reversed.

[0074] In S8004, CPU101 determines whether both the work area and the object were detected during processing S8002 and S8003. If either is not detected (S8004 / No), the process returns to S8001, the image is acquired again, and processing from S8001 onwards is executed.

[0075] If both are detected (S8004 / Yes), CPU101 executes the process of S8005.

[0076] In S8005, the region correction unit 304 performs correction processing. Here, the shape correction of the region is performed using the object region, work region, and correction reference acquired in S8002 and S8003.

[0077] In S8006, the frequency map creation unit 305 creates a frequency map using the area created in S8005. By repeatedly executing the processes from S8001 to S8005 in this flowchart, a frequency map can be created that shows the state in which work is being performed on the work area.

[0078] In the S8007 process, the frequency map created in S8006 is subjected to inverse shape correction from the correction criteria to the work area, and a heat map is created.

[0079] In S8008, the superimposed image display unit 307 superimposes a heatmap onto the image. The heatmap created in S8007 is superimposed onto the work area obtained by executing the processing in S8003 on the image acquired in S8001. The superimposed result is presented to the user via the display provided by the display unit 108.

[0080] By repeatedly executing processes S8001 through S8008, it is possible to visualize the percentage of work performed in the work area according to the work status.

[0081] Although not shown in Figure 8, the determination of whether or not to terminate the S8008 process may be made, for example, by whether the heatmap exceeds a certain value for a certain range of the work area. Alternatively, it may be determined that the process has terminated if the work area detection process and the object detection process have not been performed for a certain period of time. Alternatively, it may be determined that the process has terminated if a certain period of time has elapsed since the first object detection was performed.

[0082] In the first embodiment, when creating the frequency map, the work area and the completed area were used, with their shapes corrected based on the correction criteria. However, if the distance the part moves on the conveyor belt is short, and the size of the part detected in the image does not change even when the part is moved, the work area and completed area can also be used without shape correction.

[0083] In such cases, a frequency map is created based on the work area acquisition unit 302 and the object detection unit 303, which acquire the work area and the work area. Then, a heat map is created by adjusting the position of the created frequency map to match the position of the work area in the image acquired by the image acquisition unit 301. Furthermore, the created heat map is superimposed on the image at a position based on the work area. This allows the user to appropriately visualize the work ratio relative to the work area, even if the size of the parts detected from the image does not change, but the position of the parts does.

[0084] (Second embodiment) In the first embodiment, a method was described for correcting the frequency map by scaling the work area vertically and horizontally, and displaying a heat map that matches the work area.

[0085] In the second embodiment, shape correction will be described for cases where the shape of the work area is complex or where there is a tilt angle between the camera and the work area.

[0086] In this embodiment, the processing of the region correction unit 304, the frequency map creation unit 305, and the heat map creation unit 306 will be described. The object detection unit 301, the work area acquisition unit 302, the object detection unit 303, and the superimposed image display unit 307 are the same as those described in the first embodiment, so their description will be omitted.

[0087] The correction criteria pre-stored in the region correction unit 304 are the same as those described in the first embodiment.

[0088] The region correction unit 304 normalizes the work target region acquired by the work region acquisition unit 302, which includes the object detection region of the object detection unit 303, to the size of a certain reference region (correction standard). An example of correction using projection transformation as a normalization means will be explained using Figures 9, 10, 11, 12, and 13.

[0089] Figure 9 shows the image 900 acquired by the camera 110. The image 900 includes region 901, which is the work target area acquired by the work area acquisition unit 302. Region 901 is composed of coordinate points 902, 903, 904, and 905.

[0090] Furthermore, the captured image 900 includes a processed region 911 detected by the object detection unit 303. Region 911 is composed of coordinate points 912, 913, 914, and 915. The region correction unit 304 stores predetermined correction criteria.

[0091] The correction criteria are shown in Figure 10. Figure 10 shows the region 1000 that serves as the correction criteria, and the coordinate points 1001, 1002, 1003, and 1004 that make up region 1000.

[0092] In this embodiment, area 1000 is an area of ​​a predetermined size that is maintained internally by the system, and may be stored in the storage device 109 and read by the recording device 100 when the area correction unit 304 performs processing.

[0093] The transformation matrix for the projection transformation is calculated from the working area and the correction criteria. A system of linear equations is created by associating coordinate points 902 and 1001, 903 and 1002, 904 and 1003, and 905 and 1004, and the transformation matrix is ​​calculated from the resulting solution.

[0094] By applying the transformation matrix obtained here to the worked region, a corrected worked region aligned with the correction reference region 1000 can be obtained. The corrected worked region can be shown as in Figure 11.

[0095] In Figure 11, region 1110, which is the corrected and processed region, is plotted relative to region 1000. Region 1110 is composed of coordinate points 1111, 1112, 1113, and 1114. Coordinate points 1111, 1112, 1113, and 1114 correspond to coordinate points 912, 913, 914, and 915 of the captured image 900.

[0096] Furthermore, although this embodiment describes a method for projective transformation on a work area composed of four points, the coordinate points constituting the work area may be three or five or more. In that case, the circumscribing rectangle of the work area, the coordinate points constituting the circumscribing rectangle, and the coordinate points constituting the circumscribing rectangle of the correction reference are obtained, and a projective transformation is performed based on the obtained coordinate points to obtain a transformation matrix.

[0097] The frequency map created by the frequency map creation unit 305 is created based on a region that serves as a correction standard. The method for creating the frequency map will be explained using an example of a frequency map shown in Figure 12. Figure 12 consists of a grid 1201 that shows the grid that makes up the frequency map and a frequency region 1202 that shows the frequency information.

[0098] The region 1000 is divided into grids 1201, each consisting of predetermined blocks. It is determined whether each grid contains a worked region. Whether a worked region is included may be determined by whether the proportion of the worked region in each grid exceeds a threshold. If each grid is determined to be worked, it is counted as a detection in the frequency map.

[0099] By repeatedly acquiring captured images and creating a corrected work area and a worked area for each captured image, a frequency map is created. The resulting counted area is, for example, area 1202.

[0100] In this embodiment, the results obtained by counting were illustrated and described as shown in region 1202, but the frequency map creation unit 305 may also save them as numerical data for each grid.

[0101] For example, you could number the grids 1, 2, ... starting from the top left corner, and then store the number of detections for each grid in a corresponding format.

[0102] The heatmap creation unit 306 corrects the shape of the frequency map created by the frequency map creation unit 305 so that it matches the work area in the image acquired by the image acquisition unit 301, and then creates a heatmap.

[0103] For example, let's explain how to convert the frequency maps shown in regions 1000 and 1202 to match the work area 901 shown in Figure 9.

[0104] The transformation from region 1000 to region 901 can be performed using the inverse matrix of the transformation matrix obtained by the region correction unit 304.

[0105] The converted heatmap is created from display frequency map information adapted to the work area, obtained by converting the region 1202 of each grid in the frequency map.

[0106] Since the frequency map for display is created to fit region 901, the shape of each block needs to be scaled according to the shape of the work area. For example, in the case of a shape like the one shown in region 901, the grid size of region 1202 needs to be reduced for the upper part of the shape, and the grid size needs to be increased for the lower part of the shape.

[0107] The heatmap display information may be created with the grid size scaled up or down, or the minimum size of the display grid may be changed depending on the shape of the work area to be displayed in the heatmap.

[0108] For example, the minimum grid size of the frequency map for the heatmap may be determined to match the minimum grid size of each grid obtained when the region 1202 of each block of the frequency map is transformed using the inverse matrix of the transformation matrix acquired by the region correction unit 304. Alternatively, the frequency information may be transformed to fit a grid created to match the work area.

[0109] The method for converting frequency information will be explained based on grid 1300, which is part of a grid created to match the work area shown in Figure 13, and frequency information 1301 and 1302.

[0110] Frequency information 1301 and 1302 are parts of a grid of frequency maps converted to match the work area, obtained by converting from area 1000 to area 901.

[0111] If frequency information 1301 and 1302 have different frequency values, the value of grid 1300 may be determined according to the proportion of the area they occupy within the grid. For example, if the ratio of the areas of frequency information 1301 and 1302 is a:b, and the value of frequency information 1301 is n and the value of frequency information 1302 is m, the value obtained by (a / a+b)×m+(b / a+b)×n may be used as the frequency information for grid 1300. Alternatively, the value of the larger area between frequency information 1301 and 1302 may be adopted.

[0112] Figure 14 shows an example of a heatmap display created in the second embodiment. The captured image 1400 is shown with a color scale 1401, a work area 901, and a heatmap area 1403.

[0113] A color scale can be represented by hue or by the lightness of a color. Figure 14 shows five levels, but there is no limit to the number of levels.

[0114] Furthermore, while a projective transformation was used as the normalization means in the second embodiment, the means for achieving normalization are not limited to projective transformations. For example, free-form deformation (FFD) or affine transformation may also be used.

[0115] (Third embodiment) In the first and second embodiments, the heatmap was superimposed on the working area of ​​the captured image, but it may also be superimposed on a pre-acquired model image, for example.

[0116] At this time, the heatmap creation unit 306 deforms the frequency map obtained by the frequency map creation unit 305 to match the working area of ​​the model image. The deformation method is the same as the method described in the first and second embodiments.

[0117] Furthermore, the display method in the heatmap display may be changed according to the superimposed image. For example, if the size of the work area is less than or equal to a threshold, the transparency of the heatmap's color scale may be reduced, and if the size of the work area exceeds the threshold, the transparency of the heatmap's color scale may be increased. In addition, the thickness of the frame surrounding the area may be changed according to the size of the work area. By making the frame thinner when the work area is small and thicker when it is large, the readability of the heatmap display in the work area can be adjusted.

[0118] (Fourth embodiment) In the fourth embodiment, a mechanism for notifying the user based on the obtained frequency map and heat map will be described.

[0119] As shown in Figure 15, the storage device 100 in Figure 3 is equipped with the function of a determination unit 1508. The determination unit 1508 acquires threshold information in advance, which serves as a criterion for notifying the user. The threshold information is a numerical value representing the percentage of the region where the number of detections obtained from the frequency map exceeds a certain value relative to the entire frequency map. For example, the threshold may be set to be 70% or more of the region where the number of detections exceeds 3 relative to the frequency map.

[0120] If a threshold is exceeded, the user may be notified that the task is complete. As a means of notification, a warning lamp (not shown) may be used, or the display device 108 may display the words or icon indicating that the task is complete.

[0121] Furthermore, if the threshold is not exceeded and no work area or completed work area is detected for a certain period of time, the user may be notified.

[0122] Alternatively, the threshold can be a numerical value representing the difference between the maximum and minimum detection counts in the frequency map.

[0123] By setting thresholds for the above-mentioned work ratios and providing notification mechanisms for when the thresholds are exceeded or not exceeded, it is possible to visualize the work status and show users (workers) whether the work was done correctly or incorrectly.

[0124] Furthermore, by setting a threshold value based on the difference between the maximum and minimum detection counts in the frequency map, it is possible to indicate to the user (worker) that there are inconsistencies in the work. Note that the notification method and the method of setting the threshold may be changed as appropriate, not limited to the above.

[0125] (Other embodiments) Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0126] The disclosures herein include the following information processing devices and methods, programs, and storage media.

[0127] (Item 1) An acquisition means for acquiring images of the work area and the object, A first detection means for detecting a work area from the aforementioned image, A second detection means for detecting an object region from the aforementioned image, A frequency map creation means that creates a frequency map based on the number of times the object region is detected for each grid in the work area, A heatmap creation means that creates a heatmap by corresponding the frequency map to the work area detected by the first detection means, A display control means that superimposes the heatmap onto a display means at a position based on the work area, An information processing device characterized by comprising:

[0128] (Item 2) The information processing device according to item 1, further comprising correction means for correcting the shapes of the work area and the object area.

[0129] (Item 3) The information processing device according to item 2, characterized in that the frequency map creation means creates a frequency map based on the number of times the object region is detected for each grid in the work region corrected by the correction means.

[0130] (Item 4) The information processing apparatus according to item 2, characterized in that the correction means corrects the shapes of the work area and the object area by expanding or contracting them vertically and horizontally to match a reference area for correcting the area.

[0131] (Item 5) The information processing apparatus according to item 2, characterized in that the correction means corrects the shapes of the work area and the object area by normalizing the shapes of the work area and the object area to match a reference area for correcting the area.

[0132] (Item 6) The information processing apparatus according to item 5, characterized in that the correction means normalizes the work area and the object area to match the reference area, and the heatmap creation means converts the frequency map so that it corresponds to the work area of ​​the image acquired by the acquisition means from the reference area.

[0133] (Item 7) The information processing apparatus according to item 5 or 6, characterized in that the correction means performs the normalization by projective transformation.

[0134] (Item 8) The information processing device according to item 5 or 6, characterized in that the correction means performs the normalization by free-form deformation.

[0135] (Item 9) The information processing apparatus according to item 5 or 6, characterized in that the correction means performs the normalization by affine transformation.

[0136] (Item 10) The information processing device according to any one of items 5 to 9, characterized in that the reference area is determined based on the work area which has been imaged in advance.

[0137] (Item 11) The first detection means is characterized in that, if there are coordinate points in the work area detected in the previous detection that could not be detected in the work area detected in the current detection, and the difference between the number of coordinate points detected in the current detection and the number of coordinate points detected in the previous detection is less than or equal to a threshold, the work area detected in the previous detection is used as the work area detected in the current detection, as described in any one of items 1 to 10.

[0138] (Item 12) The information processing device according to any one of items 1 to 11, characterized in that the heat map creation means changes the size of the grid of the heat map according to the shape of the work area detected by the first detection means.

[0139] (Item 13) The information processing device according to any one of items 1 to 12, characterized in that the heatmap creation means changes at least one of the displayed color and line thickness according to the shape of the work area detected by the first detection means.

[0140] (Item 14) The information processing device according to any one of items 1 to 13, further comprising a warning means that issues a warning when the frequency and number of each grid in the frequency map do not exceed a threshold, and the work area and the object area are not detected for a certain period of time.

[0141] (Item 15) The acquisition process involves obtaining images of the work area and the object, A first detection step involves detecting a work area from the aforementioned image, A second detection step involves detecting an object region from the aforementioned image, A frequency map creation step, which creates a frequency map based on the number of times the object region is detected for each grid in the corrected work area, A heat map creation step, which creates a heat map by corresponding the frequency map to the work area detected in the first detection step, A display control step of superimposing the heatmap onto a display means at a position based on the work area, A control method for an information processing device, characterized by having the following features.

[0142] (Item 16) A program that causes a computer to execute each step of the control method for the information processing device described in item 15.

[0143] (Item 17) A computer-readable storage medium that stores a program for causing a computer to execute each step of the control method for the information processing device described in item 15.

[0144] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0145] 101: CPU, 301: Image acquisition unit, 302: Work area acquisition unit, 303: Object detection unit, 304: Area correction unit, 305: Frequency map creation unit, 306: Heat map creation unit 307 Superimposed Image Display Unit

Claims

1. Acquisition means for acquiring multiple images of a work area whose relative position to the imaging device changes, A first detection means for detecting a work area from the aforementioned plurality of images, When an object is included in the plurality of images, a second detection means for detecting the object region from the plurality of images, Correction means for correcting the shape of the work area when the size of the work area on the images differs among the plurality of images, A frequency map creation means that creates a frequency map based on the number of times the object region is detected for each grid in the work area corrected by the correction means, A heatmap creation means that creates a heatmap by corresponding the frequency map to the work area detected by the first detection means, A display control means that superimposes the heatmap onto a display means at a position based on the work area, An information processing device characterized by comprising:

2. The information processing apparatus according to claim 1, wherein the correction means further corrects the shape of the object region when the size of the work region on the images differs among the plurality of images.

3. The information processing apparatus according to claim 2, characterized in that the correction means corrects the shapes of the work area and the object area by expanding or contracting them vertically and horizontally to match a reference area for correcting the area.

4. The information processing apparatus according to claim 2, characterized in that the correction means corrects the shapes of the work area and the object area by normalizing the shapes of the work area and the object area to match a reference area for correcting the area.

5. The information processing apparatus according to claim 4, characterized in that the correction means normalizes the work area and the object area to match the reference area, and the heatmap creation means converts the frequency map so that it corresponds to the work area of ​​the image acquired by the acquisition means from the reference area.

6. The information processing apparatus according to claim 4, characterized in that the correction means performs the normalization by projection transformation.

7. The information processing apparatus according to claim 4, characterized in that the correction means performs the normalization by free-form deformation.

8. The information processing apparatus according to claim 4, characterized in that the correction means performs the normalization by affine transformation.

9. The information processing apparatus according to claim 4, characterized in that the reference area is determined based on the work area which has been imaged in advance.

10. The information processing apparatus according to claim 1, characterized in that the first detection means uses the work area detected in the previous detection as the work area detected in the current detection when there are coordinate points in the work area detected in the previous detection that could not be detected in the work area detected in the current detection, and the difference between the number of coordinate points detected in the current detection and the number of coordinate points detected in the previous detection is less than or equal to a threshold.

11. The information processing apparatus according to claim 1, characterized in that the heat map creation means changes the size of the grid of the heat map according to the shape of the work area detected by the first detection means.

12. The information processing apparatus according to claim 1, characterized in that the heat map creation means changes at least one of the displayed color and line thickness according to the shape of the work area detected by the first detection means.

13. The information processing apparatus according to claim 1, further comprising warning means for issuing a warning when the frequency and number of each grid in the frequency map do not exceed a threshold, and the work area and the object area are not detected for a certain period of time.

14. An acquisition step of acquiring multiple images of a work area whose relative position to the imaging device changes, A first detection step of detecting a work area from the aforementioned plurality of images, If an object is included in the plurality of images, a second detection step is performed to detect the object region from the plurality of images, If the size of the work area on the images differs among the multiple images, a correction step is made to correct the shape of the work area. A frequency map creation step, which creates a frequency map based on the number of times the object region is detected for each grid in the work area corrected in the correction step, A heat map creation step, which creates a heat map by corresponding the frequency map to the work area detected in the first detection step, A display control step of superimposing the heatmap onto a display means at a position based on the work area, A control method for an information processing device, characterized by having the following features.

15. A program for causing a computer to execute each step of the control method for the information processing device described in claim 14.

16. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method for the information processing apparatus described in claim 14.

17. Acquisition means for acquiring an image of a work area and an object, A first detection means for detecting a work area from the aforementioned image, A second detection means for detecting an object region from the aforementioned image, Correction means for correcting the shapes of the work area and the object area, A frequency map creation means that creates a frequency map based on the number of times the object region is detected for each grid in the work area, A heatmap creation means that creates a heatmap by corresponding the frequency map to the work area detected by the first detection means, The system includes a display control means for displaying the heatmap superimposed on a display means at a position based on the work area, The correction means is characterized by correcting the shapes of the work area and the object area by normalizing the shapes of the work area and the object area to match a reference area for correcting the area.

18. An acquisition step of acquiring an image of the work area and an object, A first detection step involves detecting a work area from the aforementioned image, A second detection step involves detecting an object region from the aforementioned image, A correction step for correcting the shapes of the work area and the object area, A frequency map creation step, which creates a frequency map based on the number of times the object region is detected for each grid in the work area, A heat map creation step, which creates a heat map by corresponding the frequency map to the work area detected in the first detection step, The system includes a display control step that causes the heatmap to be superimposed on a display means at a position based on the work area and displayed on the display means, A control method for an information processing device, characterized in that, in the correction step, the shapes of the work area and the object area are corrected by normalizing the shape of the work area and the shape of the object area to match a reference area for correcting the area.

Images