Image processing system, image processing device, and image processing method

Abstract

This image processing system comprises an imaging device and a computational device. The imaging device generates wide-angle image data representing at least a situation in which a worker performs work on an object. The computational device acquires, on the basis of the wide-angle image data and reference data pertaining to work-associated movement of a detection target, which includes the worker and / or the object, enlarged image data representing, in an enlarged manner, a first portion included in an imaging range of the imaging device that corresponds to the wide-angle image data. The enlarged image data is data representing at least a part of the detection target.

Description

Image processing system, image processing device, and image processing method

[0001] The present disclosure relates to an image processing system, an image processing device, and an image processing method for processing image data.

[0002] Patent document 1 describes an image processing system that has a wide-angle camera, a close-up camera that captures part of the imaging range of the wide-angle camera, and a drive mirror, and the drive mirror moves the imaging range of the close-up camera.

[0003] International Publication No. 2018 / 138349

[0004] An image processing system according to an embodiment of the present invention includes an imaging device that generates wide-angle image data that at least represents a situation in which a worker is performing work on an object, and a computing device that acquires enlarged image data that represents an enlarged first portion included in an imaging range captured by the imaging device corresponding to the wide-angle image data, based on the wide-angle image data and reference data related to the movement of a detection target, which includes at least one of the worker and the object, during the work. The enlarged image data represents at least a portion of the detection target.

[0005] An image processing system according to another embodiment of the present invention includes an imaging device that generates wide-angle image data representing at least a situation in which a worker is performing work on an object, and a calculation device that acquires, based on the wide-angle image data, enlarged image data representing a first portion included in an imaging range captured by the imaging device corresponding to the wide-angle image data. The wide-angle image data represents the worker's face, and the enlarged image data represents an enlarged version of at least a portion of the worker's face as the first portion, and the calculation device identifies an object in the worker's line of sight based on the enlarged image data.

[0006] An image processing system according to another embodiment of the present invention includes an imaging device that generates wide-angle image data that at least represents a situation in which at least one of a worker and a work device is performing work on an object, and a computing device that, based on the wide-angle image data, obtains enlarged image data that represents an enlarged first portion included in the imaging range of the imaging device that corresponds to the wide-angle image data, and, based on the enlarged image data, performs at least one of notifying the worker to change the work on the object and outputting a work control signal to the work device that controls the work on the object.

[0007] An image processing system according to another embodiment of the present invention includes an imaging device that generates wide-angle image data in a time series, the wide-angle image data representing a situation including an object, and a calculation device that, based on the wide-angle image data, acquires enlarged image data representing an enlarged first portion included in an imaging range of the imaging device corresponding to the wide-angle image data, and creates a moving image including at least one of a wide-angle moving image using the wide-angle image data and an enlarged moving image using the enlarged image data. The calculation device detects, from the wide-angle image data, first trigger information that sets a start timing of the moving image among trigger information that sets at least one of a start timing and an end timing of the moving image, detects, from the wide-angle image data after the start timing, second trigger information that sets an end timing of the moving image among the trigger information, and generates a moving image using at least one of the wide-angle image data and the enlarged image data from the start timing to the end timing.

[0008] An image processing system according to another embodiment of the present invention includes an imaging device that generates, in a time series, wide-angle image data representing a situation including an object, first enlarged image data that enlarges and represents a first portion included in an imaging range corresponding to the wide-angle image data, and second enlarged image data that enlarges and represents a second portion that is included in an imaging range corresponding to the wide-angle image data and is different from the first portion, and a calculation device that determines, based on information about the first portion and information about the second portion, at least one of a first frame rate for generating the first enlarged image data in a time series and a second frame rate for generating the second enlarged image data in a time series, and creates a first enlarged video using the first enlarged image data and a second enlarged video using the second enlarged image data.

[0009] An image processing system according to another embodiment of the present invention includes an imaging device that generates, in a time series, wide-angle image data representing a situation including an object, first enlarged image data that enlarges and represents a first portion included in an imaging range corresponding to the wide-angle image data, and second enlarged image data that enlarges and represents a second portion different from the first portion included in the imaging range corresponding to the wide-angle image data, and an arithmetic device that creates a first enlarged video using the latest first enlarged image data from the first enlarged image data generated in a time series at a plurality of first image selection timings each set at a first time interval, and creates a second enlarged video using the latest second enlarged image data from the second enlarged image data generated in a time series at a plurality of second image selection timings each set at a second time interval.

[0010] An image processing system according to another embodiment of the present invention includes an imaging device that acquires wide-angle image data representing a situation including an object, and a computing device that acquires enlarged image data representing a first portion including at least a part of the object based on an imaging range captured by the imaging device corresponding to the wide-angle image data. The computing device sets an object recognition area within the wide-angle image data that is used to acquire the enlarged image data, and acquires the enlarged image data if the first portion is included in the object recognition area.

[0011] An image processing device according to an embodiment of the present invention includes a processor configured to generate still images representing the movement of at least one of a plurality of objects and a worker within a predetermined period of time, based on wide-angle image data representing the status of a plurality of objects each moving within a predetermined period of time and the status of a worker performing a process involving a plurality of tasks while moving sequentially within a predetermined area with respect to each of the plurality of objects.

[0012] An image processing method according to an embodiment of the present invention includes: acquiring, by an imaging device, first wide-angle image data including an object at a first timing; acquiring, by a calculation device, first enlarged image data based on the first wide-angle image data, which enlarges and represents a first portion including at least a part of the object at the first timing; acquiring, by the imaging device, second wide-angle image data including the object at a second timing after the first timing; and, if it is determined using the second wide-angle image data that the amount of change from the position of the first portion at the first timing to the position of the first portion at the second timing is within a change threshold, acquiring, by the calculation device, second enlarged image data which enlarges and represents the first portion at the second timing.

[0013] FIG. 1 is a diagram illustrating an overview of the operation of an image processing system. FIG. 2 is a diagram illustrating the schematic configuration of an imaging device and a computing device. FIG. 3 is a first example of wide-angle image data and enlarged image data. FIG. 4 is a second example of wide-angle image data and enlarged image data. FIG. 5 is a third example of wide-angle image data and enlarged image data. FIG. 6 is a diagram illustrating capturing an image of a worker's face using a mirror. FIG. 7 is a diagram illustrating creating a video using enlarged image data. FIG. 8(A) is a diagram illustrating a first example of image selection using a search range, FIG. 8(B) is a diagram illustrating a second example of image selection using a search range, and FIG. 8(C) is a diagram illustrating a third example of image selection using a search range. FIG. 9(A) is an example of a screen displaying both wide-angle video and enlarged video, FIG. 9(B) is an example of wide-angle image data representing an object being transported, and FIG. 9(C) is an example of enlarged image data representing an object being transported. FIG. 10 is a diagram illustrating the movement of a worker within a predetermined area. Fig. 11(A) is a first example of a still image showing the movement of a worker, Fig. 11(B) is a second example of a still image showing the movement of a worker, and Fig. 11(C) is a third example of a still image showing the movement of a worker. Fig. 12 is an example of an image showing the duration of stay by process. Fig. 13 is an example of detailed image information. Fig. 14 is an example of a heat map.

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] (1) Configuration of Image Processing System SYS (1-1) Overall Configuration of Image Processing System SYS FIG. 1 is a schematic diagram of an image processing system SYS according to an embodiment of the present invention. The image processing system SYS includes an imaging device 100 and a computing device 170. The imaging device 100 generates wide-angle image data WD representing a situation in which an operator OP performs work on an object TG. The situation in which the operator OP performs work on the object TG can also be referred to as a work environment OE. The work environment OE may include a work apparatus 200-1 and a work apparatus 200-2 (hereinafter collectively referred to as "work apparatus 200") that perform work on the object TG. The situation in which at least one of the operator OP and the work apparatus 200 performs work on the object TG can also be referred to as a work environment OE. The working device 200 may have a working member such as an end effector for performing work on the target object TG, and may further have a movable member such as a robot arm or a platform for moving the working device 200 itself. The number of working devices 200 included in the working environment OE is not limited to two, but may be one, or may be three or more. The working environment OE may include a mobile device 300 that can move within the working environment OE. The mobile device 300 can be said to be mobile around the target object TG, mobile around the worker OP, or mobile around the working device 200. The number of mobile devices 300 included in the working environment OE is not limited to one, but may be two or more. The working device 200 may be mobile within the working environment OE. The working device 200 that can move within the working environment OE can also be said to be a mobile device 300.

[0016] The arithmetic device 170 acquires enlarged image data MD that enlarges and represents a first portion of the wide-angle image data WD. The enlarged image data MD can also be considered data that enlarges and represents a first portion included in the imaging range of the imaging device 100 that corresponds to the wide-angle image data WD. The arithmetic device 170 may acquire the enlarged image data MD generated by the imaging device 100. The first portion may include the target object TG. The first portion may include at least one of the eyes, face, hands, feet, and torso of the worker OP. The first portion may include the working device 200 or the moving device 300.

[0017] The arithmetic device 170 may cause the output device 174 to execute output based on at least one of the wide-angle image data WD and the enlarged image data MD. The output device 174 may include any of a display device 174-1, a speaker 174-2, and an indicator light 174-3. The output device 174 may also include any of a wearable terminal (not shown) and smart glasses (not shown). The wearable terminal and smart glasses have displays and can visually display various information to the user (worker OP) wearing them. The wearable terminal can also output sounds and vibrations to the user (worker OP) wearing them. By causing the output device 174 to execute output, the arithmetic device 170 can prompt the worker OP to change the work performed on the target object TG. By causing the output device 174 to execute output, the arithmetic device 170 can notify the worker OP of the pass / fail determination result of the work performed on the target object TG. The calculation device 170 may output a work control signal for controlling work on the target object TG to the working device 200 based on at least one of the wide-angle image data WD and the enlarged image data MD. The calculation device 170 may output a movement control signal for the working device 200 or the moving device 300 to the working device 200 or the moving device 300 based on at least one of the wide-angle image data WD and the enlarged image data MD. The movement control signal may be a control signal for movement of a part of the working device 200 (e.g., a change in at least one of the position and attitude of a movable member) or movement of the moving device 300 (e.g., a change in at least one of the position and attitude of the moving device 300).

[0018] The object TG may include a plurality of objects. The plurality of objects may be a plurality of types of objects having different appearance characteristics such as size, shape, color, etc. The object TG may move within the working environment OE. In the example shown in FIG. 1, the object TG moves within the working environment OE by being conveyed by a conveyor CV.

[0019] (1-2) Configuration of the Imaging Device 100 FIG. 2 is a schematic configuration diagram of the imaging device 100 and the arithmetic device 170. The imaging device 100 includes an imaging optical system 110, a light splitting member 120, a magnifying optical system 130, a first imaging device 140, a second imaging device 150, and a change device 160. The imaging optical system 110, the light splitting member 120, the magnifying optical system 130, the first imaging device 140, the second imaging device 150, and the change device 160 are housed and held in a housing 101. The arithmetic device 170 is connected to the imaging device 100. The arithmetic device 170 is also connected to an output device 174. The imaging optical system 110 is an example of a first optical system, and the magnifying optical system 130 is an example of a second optical system.

[0020] In this embodiment, the optical axis of the first magnifying optical system 131 of the imaging optical system 110 and the magnifying optical system 130 is referred to as the first optical axis AX1. The optical axis of the second magnifying optical system 132 of the magnifying optical system 130 is referred to as the second optical axis AX2. The second optical axis AX2 may intersect with the first optical axis AX1. The second optical axis AX2 may be perpendicular to the first optical axis AX1. The direction along the first optical axis AX1 is referred to as the Y direction. The direction along the second optical axis AX2 is referred to as the Z direction. The direction perpendicular to the Y direction and the Z direction is referred to as the X direction. In other words, the X direction, the Y direction, and the Z direction are perpendicular to each other. For example, as shown in FIG. 2 , the Z direction may be vertical, and the X direction and the Y direction may be horizontal.

[0021] The imaging optical system 110 includes a front lens 111, a rear lens 112, and an aperture stop 113. The imaging optical system 110 may be telecentric on the side of the light splitting member 120 (the side of the intermediate image Imd described below). The front lens 111 is disposed farther from the light splitting member 120 than the aperture stop 113. The rear lens 112 is disposed closer to the light splitting member 120 than the aperture stop 113. In FIG. 2 , the front lens 111 and the rear lens 112 are each schematically shown as a single lens. The front lens 111 and the rear lens 112 may each be composed of a single lens or multiple lenses. The front lens 111 and the rear lens 112 may each include one or more lenses as well as optical elements other than lenses. The front lens 111 may include a focusing lens that is movable along the optical axis during focusing. The light splitting member 120 is not limited to a two-branching type, and may be a three-branching type or other type that branches multiple times. By using a three-branching light splitting member, it is possible to obtain enlarged image data at multiple magnifications.

[0022] The imaging optical system 110 may be interchangeable. The interchangeable imaging optical system 110 may be, for example, at least one of a fixed focal length lens, a telecentric lens, a zoom lens, and a fisheye lens. In other words, the imaging optical system 110 may be a fixed focal length lens, a telecentric lens, a zoom lens, or a fisheye lens. Furthermore, the interchangeable imaging optical system 110 may be made up of multiple types of fixed focal lengths.

[0023] The imaging optical system 110 does not have to be interchangeable. In this case, the imaging optical system 110 may be at least one of a single-focus lens and a fisheye lens. The maximum angle of view of the imaging optical system 110 (i.e., the maximum angle of view on the object side of the image captured by the first image capture device 140) may be 170° or more. The maximum angle of view of the imaging optical system 110 may be less than 170°. For example, the maximum angle of view of the imaging optical system 110 may be 160°.

[0024] The light splitting member 120 is disposed between the imaging optical system 110 and the magnifying optical system 130. The light splitting member 120 splits the light LT that has passed through the imaging optical system 110, and directs one light beam to the first image capture device 140 and the other light beam to the magnifying optical system 130. For example, the light splitting member 120 may perform amplitude splitting on the light LT that has passed through the imaging optical system 110. Such a light splitting member 120 may also be referred to as a half mirror. Specifically, the light splitting member 120 may reflect a portion of the light LT that has passed through the imaging optical system 110 in the +Z direction toward the first image capture device 140 and transmit the remaining light. The half mirror may be a prism type or a flat type. The ratio of the transmittance to the reflectance of the half mirror may be 1:1 or another ratio. For example, this ratio may be 2:1.

[0025] The light splitting member 120 may split the wavelength of the light LT that has passed through the imaging optical system 110. Such a light splitting member 120 may also be referred to as a dichroic mirror. Specifically, the light splitting member 120 may reflect light in the near-infrared wavelength range, of the light LT that has passed through the imaging optical system 110, in the +Z direction toward the first image capture device 140, and transmit light in other wavelength ranges (visible light). The light splitting member 120 may transmit light in the near-infrared wavelength range, of the light LT that has passed through the imaging optical system 110, and reflect light in other wavelength ranges (visible light) in the +Z direction toward the first image capture device 140.

[0026] The light splitting member 120 may polarize and split the light LT that has passed through the imaging optical system 110. Such a light splitting member 120 can also be called a polarizing beam splitter. The light splitting member 120 may be disposed midway through the imaging optical system 110. In this case, the light splitting member 120 splits the light LT that has passed through a portion of the imaging optical system 110. For example, a rear lens 112 may be disposed between the light splitting member 120 and the first image Im1 and between the light splitting member 120 and the intermediate image Imd.

[0027] Of the light LT that has passed through at least a portion of the imaging optical system 110, the first image Im1 is formed by light that is reflected by the light splitting member 120. In other words, the first image Im1 is formed by one of the light beams split by the light splitting member 120. Furthermore, of the light LT that has passed through the imaging optical system 110, the intermediate image Imd is formed by light that has passed through the light splitting member 120. In other words, the intermediate image Imd is formed by the other of the light beams split by the light splitting member 120. The intermediate image Imd is also referred to as a third image. Here, the position where the intermediate image Imd is formed may be conjugate to the position where the first image Im1 is formed. In other words, the intermediate image Imd may be an image conjugate to the first image Im1. The first image Im1 formed by one of the light beams split by the light splitting member 120 can also be referred to as an intermediate image. Note that the light LT that has passed through at least a part of the imaging optical system 110 is split by the light splitting member 120, and the first image Im1 and the intermediate image Imd are formed by the respective split light beams, so it can also be said that the light splitting member 120 contributes to image formation. Therefore, the light splitting member 120 may be a part of the imaging optical system 110.

[0028] The magnifying optical system 130 forms a second image Im2 by enlarging and re-imaging a portion of the intermediate image Imd formed by the light LT that has passed through the imaging optical system 110. In other words, the second image Im2 is formed via the magnifying optical system 130 using the other light split by the light splitting member 120. As described above, the position at which the intermediate image Imd is formed may be conjugate with the position at which the first image Im1 is formed. By having the magnifying optical system 130 enlarge and re-imaging a portion of the intermediate image Imd, an effect equivalent to re-imaging a portion of the first image Im1 (a portion at the same relative position as the intermediate image Imd) can be obtained. In other words, the magnifying optical system 130 enlarges and re-imaging a portion of the intermediate image Imd that corresponds to a portion of the first image Im1.

[0029] The magnifying optical system 130 may be telecentric on the intermediate image Imd (beam splitting member 120) side. This suppresses divergence of the chief ray near the intermediate image Imd, allowing the aperture of the magnifying optical system 130 to be reduced. This allows the image processing system SYS to be miniaturized and manufacturing costs to be reduced. Furthermore, it is possible to suppress variations in the angle of incidence of light incident on the beam splitting member 120 (beam splitting surface). In the example shown in FIG. 2 , both the imaging optical system 110 and the magnifying optical system 130 are telecentric on the intermediate image Imd (beam splitting member 120) side, but this is not a limitation. For example, only one of the imaging optical system 110 and the magnifying optical system 130 may be telecentric on the intermediate image Imd (beam splitting member 120) side.

[0030] The magnifying optical system 130 includes a first magnifying optical system 131 and a second magnifying optical system 132. Light that has passed through the light splitting member 120 is incident on the first magnifying optical system 131. In other words, the other light split by the light splitting member 120 is incident on the first magnifying optical system 131. Light from the first magnifying optical system 131 is incident on the second magnifying optical system 132. In FIG. 2 , the first magnifying optical system 131 and the second magnifying optical system 132 are each schematically shown as a single lens. The first magnifying optical system 131 and the second magnifying optical system 132 may be composed of one or more lenses. Each of the first magnifying optical system 131 and the second magnifying optical system 132 may include optical elements other than lenses in addition to one or more lenses.

[0031] The optical elements constituting the first magnifying optical system 131 and the second magnifying optical system 132 may be housed in a lens barrel. In this case, the inner circumferential surface of the lens barrel may be anti-reflection treated. This prevents flare caused by light reflection on the inner surface of the lens barrel. The anti-reflection treatment may be achieved, for example, by arranging flocked paper on the inner circumferential surface. The first magnifying optical system 131 and the second magnifying optical system 132 may also include one or more annular masks. This prevents light reflected on the inner surface of the lens barrel from being blocked by the mask, thereby preventing flare caused by light reflection on the inner surface of the lens barrel. In this case, the inner diameter of the lens barrel may be larger than the outer diameter of the lens. This prevents light reflected on the inner surface of the lens barrel from leaking outside the mask due to tolerances in the outer diameter of the mask, thereby preventing such light from entering the lens and preventing flare.

[0032] The optical axis (second optical axis AX2) of the second magnifying optical system 132 may be perpendicular to the optical axis (first optical axis AX1) of the first magnifying optical system 131. The optical axis of the second magnifying optical system 132 may intersect with the optical axis of the first magnifying optical system 131 at an angle other than 90 degrees. For example, the optical axis of the second magnifying optical system 132 may intersect with the optical axis of the first magnifying optical system 131 within an angle range of 90 degrees ± 5 degrees. The reflecting surface 162 of the reflecting member 161 included in the adjustment device 160 is disposed at or near the intersection position where the optical axis of the first magnifying optical system 131 intersects with the optical axis of the second magnifying optical system 132. The intersection position where the optical axis of the first magnifying optical system 131 intersects with the optical axis of the second magnifying optical system 132 is also referred to as the intersection position where the optical axis of the first magnifying optical system 131 intersects with the optical axis of the second magnifying optical system 132. The vicinity of the intersection position may be, for example, a position different from the intersection position where the reflecting surface 162 of the reflecting member 161 can reflect light from the first magnifying optical system 131 toward the second magnifying optical system 132. The second magnifying optical system 132 forms the second image Im2 with the light from the first magnifying optical system 131 reflected by the reflecting surface 162 of the reflecting member 161. In other words, it can be said that the second image Im2 is formed by the first magnifying optical system 131 and the second magnifying optical system 132. It can also be said that the magnifying optical system 130 includes the reflecting member 161 in addition to the first magnifying optical system 131 and the second magnifying optical system 132.

[0033] The second magnifying optical system 132 may be interchangeable. The interchangeable second magnifying optical system 132 may be, for example, an optical system with a fixed magnification or a variable magnification optical system (zoom lens). In other words, the second magnifying optical system 132 may be an optical system with a fixed magnification or a variable magnification optical system (zoom lens). Furthermore, the interchangeable second magnifying optical system 132 may be made up of multiple types of optical systems with different magnifications.

[0034] The second magnification optical system 132 does not have to be interchangeable. In this case, too, the second magnification optical system 132 may be a fixed magnification optical system or a variable magnification optical system. Because the second magnification optical system 132 is a fixed magnification optical system or a variable magnification optical system, the magnification optical system 130 can also be said to be a fixed magnification optical system or a variable magnification optical system. The magnification optical system 130 may be configured to re-image the second image Im2 by reducing a portion of the intermediate image Imd, or may be configured to re-image the second image Im2 at the same size as the portion of the intermediate image Imd. When the magnification optical system 130 re-images the second image Im2 without enlarging the portion of the intermediate image Imd, the second imaging device 150 may enlarge the second image Im2 to generate enlarged image data (digital zoom).

[0035] The first imaging device 140 includes a first imaging element 141 that captures a first image Im1. The first imaging element 141 may be a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The first imaging element 141 may also be an infrared image sensor capable of capturing infrared images. In this case, the first imaging element 141 may be a near-infrared image sensor capable of capturing near-infrared images, or a mid-infrared image sensor capable of capturing mid-infrared images. The first imaging element 141 may also be an event-based vision sensor. For example, if the light reflected by the light dividing member 120 is visible light, the first imaging element 141 may be a CMOS image sensor or an event-based vision sensor. If the light reflected by the light dividing member 120 includes light in the near-infrared wavelength range, the first imaging element 141 may be a near-infrared image sensor. If the light reflected by the light dividing member 120 includes light in the mid-infrared wavelength range, the first imaging element 141 may be a mid-infrared image sensor.

[0036] The first image sensor 141 is disposed at or near the position of the image plane where the first image Im1 is formed. The vicinity of the position of the image plane where the first image Im1 is formed is, for example, near the range where the first image sensor 141 can capture the first image Im1 so that the object TG can be identified in the image of the first image Im1.

[0037] A plurality of pixels are two-dimensionally arranged in the imaging region of the first imaging element 141. The plurality of pixels are arranged at a predetermined arrangement pitch in the X direction and the Y direction. For example, 1,000 or more pixels may be arranged in each of the X direction and the Y direction. The pixels of the first imaging element 141 perform photoelectric conversion of light that is reflected by the light splitting member 120 and enters the first imaging element 141. The first imaging device 140 outputs image data of the first image Im1 generated based on the photoelectric conversion at each pixel of the first imaging element 141 to the calculation device 170.

[0038] The second imaging device 150 includes a second imaging element 151 that captures the second image Im2. The second imaging element 151 may be a CMOS image sensor. The second imaging element 151 may also be an infrared image sensor. In this case, the second imaging element 151 may be a near-infrared image sensor or a mid-infrared image sensor. The second imaging element 151 may also be an event-based vision sensor. For example, if the light passing through the light dividing member 120 is visible light, the second imaging element 151 may be a CMOS image sensor or an event-based vision sensor. If the light passing through the light dividing member 120 includes light in the near-infrared wavelength range, the second imaging element 151 may be a near-infrared image sensor.

[0039] When the light reflected by and the light transmitted through the light dividing member 120 are both visible light, the first imaging element 141 may be an event-based vision sensor, and the second imaging element 151 may be a CMOS image sensor. When the light reflected by the light dividing member 120 is visible light, and the light transmitted through the light dividing member 120 includes light in the near-infrared wavelength range, the first imaging element 141 may be an event-based vision sensor, and the second imaging element 151 may be a near-infrared image sensor. When the light reflected by and the light transmitted through the light dividing member 120 both include light in the near-infrared wavelength range, the first imaging element 141 and the second imaging element 151 may be near-infrared image sensors. When the light reflected by and the light transmitted through the light dividing member 120 both include light in the mid-infrared wavelength range, the first imaging element 141 and the second imaging element 151 may be mid-infrared image sensors.

[0040] The second image sensor 151 is disposed at or near the position of the image plane where the second image Im2 is formed. The vicinity of the position of the image plane where the second image Im2 is formed is, for example, near the range where the second image sensor 151 can capture the second image Im2 so that the object TG can be identified in the image of the second image Im2.

[0041] A plurality of pixels are arranged two-dimensionally in the imaging region of the second imaging element 151. The plurality of pixels are arranged at a predetermined arrangement pitch in the X direction and the Y direction. For example, 1000 or more pixels may be arranged in each of the X direction and the Y direction. The pixels of the second imaging element 151 perform photoelectric conversion on light that passes through the light splitting member 120, passes through the magnifying optical system 130, and enters the second imaging element 151. The second imaging device 150 outputs image data of the second image Im2 generated based on the photoelectric conversion at each pixel of the second imaging element 151 to the calculation device 170.

[0042] The pixel arrangement pitch of the second imaging element 151 may be smaller than the pixel arrangement pitch of the first imaging element 141. In other words, the number of pixels per unit area of ​​the effective area (area where pixels are arranged) of the second imaging element 151 may be larger (higher density) than the number of pixels per unit area of ​​the effective area (area where pixels are arranged) of the first imaging element 141. In other words, the second imaging element 151 may be an imaging element with a higher resolution than the first imaging element 141. The pixel arrangement pitch of the second imaging element 151 may be equal to or smaller than the pixel arrangement pitch of the first imaging element 141. For example, the area of ​​the effective area of ​​the second imaging element 151 may be smaller than the area of ​​the effective area of ​​the first imaging element 141. The area of ​​the effective area of ​​the second imaging element 151 may be equal to or larger than the area of ​​the effective area of ​​the first imaging element 141.

[0043] The change device 160 includes a reflecting member 161 having a reflecting surface 162 and a driving device 163. The change device 160 can also be considered a change device that changes the optical path of the other light (light transmitted through the light splitting member 120) split by the light splitting member 120. As described above, the reflecting surface 162 of the reflecting member 161 may be disposed at or near the intersection of the optical axis (first optical axis AX1) of the first magnifying optical system 131 and the optical axis (second optical axis AX2) of the second magnifying optical system 132. The reflecting surface 162 of the reflecting member 161 may also be disposed at the pupil position or pupil conjugate position of the optical system consisting of the imaging optical system 110 and the first magnifying optical system 131. This allows the area of ​​the reflecting surface 162 of the reflecting member 161 to be reduced. The reflecting surface 162 of the reflecting member 161 reflects at least a portion of the light that has passed through the light splitting member 120 and the first magnifying optical system 131 toward the second magnifying optical system 132. In other words, the reflecting surface 162 of the reflecting member 161 reflects at least a part of the other light split by the light splitting member 120 toward the second imaging device 150 .

[0044] The modification device 160 may be configured to modify at least one of the direction and the position of the second imaging device 150. Such a modification device 160 may not include the reflecting member 161. Such a modification device 160 can also be considered a modification member that modifies the optical path of the other light split by the light splitting member 120 (light that has passed through the light splitting member 120).

[0045] The driving device 163 includes, for example, a voice coil motor (VCM) and rotates the reflecting member 161 in accordance with a control signal supplied from the computing device 170. The driving device 163 may rotate the reflecting member 161 around two rotation axes, one extending in the X direction and the other extending in the Y direction. In other words, the changing device 160 may be a two-axis gimbal mirror that can rotate the reflecting member 161 around two rotation axes. This allows the driving device 163 to rotate the reflecting member 161 to an angle that reflects light traveling along the optical axis (first optical axis AX1) of the first magnifying optical system 131 toward the optical axis (second optical axis AX2) of the second magnifying optical system 132, and an angle that reflects light traveling off the optical axis of the first magnifying optical system 131 toward the optical axis of the second magnifying optical system 132. The changing device 160 is not limited to a two-axis gimbal mirror and may be another device. For example, the modification device 160 may have two reflecting members 161. In this case, the driving device 163 may rotate each of the two reflecting members 161 around a different rotation axis. The driving device 163 may also rotate each of the two reflecting members 161 around rotation axes that are perpendicular to each other. In other words, the modification device 160 may be a two-axis galvanometer mirror. The modification device 160 is not limited to a device that rotates the reflecting member 161 around two rotation axes, but may also be a device that rotates the reflecting member 161 around a single rotation axis (for example, a single-axis gimbal mirror or a single-axis galvanometer mirror) or a device that rotates the reflecting member 161 around three rotation axes (for example, a three-axis gimbal mirror or a three-axis galvanometer mirror).

[0046] The arithmetic device 170 is, for example, a PC (Personal Computer) etc. The arithmetic device 170 can also be called a control device because it is a device that controls other devices such as the imaging device 100 and the output device 174. The arithmetic device 170 has a memory 171, an interface 172, and a processor 173.

[0047] The memory 171 is an example of a storage device and stores data and programs. The memory 171 is, for example, a semiconductor memory, a magnetic disk, etc. The memory 171 stores an operating system program, a driver program, an application program, and various data used in the operation of the image processing system SYS, which are used in processing by the processor 173.

[0048] The interface 172 is a device that enables the arithmetic device 170 to communicate with other devices such as the imaging device 100 and the output device 174. The interface 172 is, for example, a serial interface or a communication interface. The communication interface may be a wired interface such as a wired LAN, or a wireless interface such as a wireless LAN or a wireless WAN. The interface 172 may also be a video input / output interface. The interface 172 outputs data supplied from the processor 173 to other devices such as the imaging device 100 and the output device 174.

[0049] The processor 173 is an example of a processing unit, and performs overall control of the operation of the arithmetic device 170. The processor 173 is, for example, a central processing unit (CPU), a digital signal processor (DSP), a large scale integration (LSI), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). The processor may also be referred to as a group of circuits. The processor 173 executes various processes based on programs stored in the memory 171.

[0050] The processor 173 controls the first imaging device 140 to generate image data at each predetermined imaging cycle. The image data generated by the first imaging device 140 is wide-angle image data WD that represents the situation in which the worker OP performs work on the target object TG. The wide-angle image data WD includes multiple still image data corresponding to multiple times. The processor 173 acquires the wide-angle image data WD generated by the first imaging device 140 and stores it in the memory 171 in association with the time at which the wide-angle image data WD was generated.

[0051] The processor 173 controls the second imaging device 150 to generate image data at each predetermined imaging cycle. The image data generated by the second imaging device 150 is data obtained by enlarging a portion represented by the wide-angle image data, such as enlarged image data MD obtained by enlarging an object TG. The enlarged image data MD includes multiple still image data corresponding to multiple times. The processor 173 acquires the enlarged image data MD generated by the second imaging device 150 and stores the enlarged image data MD in memory 171 in association with the time at which the enlarged image data MD was generated. The second imaging device 150 can acquire enlarged image data MD of multiple locations in the wide-angle image data by controlling the modification device 160.

[0052] The processor 173 controls the change device 160 based on the wide-angle image data WD. The processor 173 supplies a control signal to the drive device 163 to drive the drive device 163 and rotate the reflecting member 161 so that an image of a portion (first portion) of the work environment OE represented in the wide-angle image data WD is formed on the imaging surface of the second imaging element 151 via the magnifying optical system 130. The first portion is, for example, a detection target. For example, the processor 173 controls the rotation angle of the reflecting member 161 based on position information of the first detection target generated from the wide-angle image data WD so that an image of the first detection target is formed on the imaging surface of the second imaging element 151 via the magnifying optical system 130. Furthermore, the processor 173 controls the rotation angle of the reflecting member 161 based on position information of the second detection target generated from the wide-angle image data WD so that an image of the second detection target is formed on the imaging surface of the second imaging element 151 via the magnifying optical system 130. The portion of the second detection target represented in the wide-angle image data WD can also be referred to as a second portion different from the first portion. If a table representing the relationship between the position information of the detection target and the rotation angle of the reflecting member 161 is stored in advance in the memory 171, the processor 173 can determine the rotation angle of the reflecting member 161 by referencing the table. The first portion is not limited to the detection target. For example, the first portion may be a change area that differs from a reference environment, which is a normal work environment OE in which a worker OP works on an object TG. The reference environment may be stored in advance in the memory 171. The reference environment stored in advance in the memory 171 may also be referred to as reference data. In this case, the reference environment may be stored in the same data format as the wide-angle image data WD. Multiple change areas may also be detected from the wide-angle image data WD. In this case, one of the multiple change areas may be referred to as the first portion, and another change area other than the one change area may be referred to as the second portion. In generating the identification information described above, the processor 173 may generate first identification information that identifies the first portion and second identification information that identifies the second portion. In the following, when there is no need to distinguish between the first part and the second part, they will also be collectively referred to as the "first part."

[0053] The processor 173 may detect, as the first portion, a portion of the work environment OE that corresponds to detection reference information that is stored in advance in memory as reference data. The detection reference information may be information that represents an event that is desired to be detected in the work environment OE. The event that is desired to be detected in the work environment OE may be, for example, an action of the worker OP raising his / her hand to a predetermined height, an action of the worker OP hiding a specific hand, or a specific object no longer appearing in the wide-angle image data.

[0054] The processor 173 may use the wide-angle image data WD to create a wide-angle video. For example, the processor 173 creates a wide-angle video in a predetermined video format, such as MP4 or MPEG, so that at least some of the still image data pieces corresponding to multiple time points included in the wide-angle image data WD are displayed in the order in which they were generated, and stores the created wide-angle video in the memory 171. The processor 173 may also create a wide-angle still image in a predetermined still image format, such as JPEG or PNG, from any of the still image data pieces included in the wide-angle image data WD and store the created wide-angle still image in the memory 171. At this time, the processor 173 may store the wide-angle image data WD acquired from the imaging device 100 in the memory 171 and create the wide-angle video using the stored wide-angle image data WD. The processor 173 may also create an enlarged video using the enlarged image MD. The method for creating an enlarged video will be described later.

[0055] The imaging device 100 may further include an adjustment device. The adjustment device can be disposed in the housing 101 or attached externally, and is used to adjust the shooting conditions of the imaging device 100. The adjustment device may be, for example, a focus mechanism that adjusts the focus of at least one of the imaging optical system 110 and the magnifying optical system 130, or a light attached to the housing 101 that can adjust the illumination conditions. The focus mechanism is preferably disposed so that the position of the lens of the imaging optical system 110 can be adjusted. Alternatively, both the focus mechanism and the light may be disposed in the housing 101.

[0056] (1-3) Information Acquired from Wide-Angle Image Data WD and Enlarged Image Data MD Image data including the wide-angle image data WD and enlarged image data MD includes at least one of information related to the work performed by the worker OP or the work device 200 as shown in Fig. 3, and identification information related to the object represented in the wide-angle image data WD and enlarged image data MD, including item information, number information, position information, or posture information of the worker OP, target object TG, work device 200, or mobile device 300. Processor 173 acquires the identification information from the image data.

[0057] The item information includes, for example, product information of the detection target, and barcode and character information written on the detection target.

[0058] The number information is information regarding the number of detection targets represented in the wide-angle image data WD and the enlarged image data MD. The number information may be, for example, the number of detection targets detected according to the model number or product number. The position information is information regarding the position of the detection targets represented in the wide-angle image data WD and the enlarged image data MD.

[0059] The position information may be position information of the detection target in the first imaging element 141 or the second imaging element 151. For example, the processor 173 detects the detection target from the wide-angle image data WD using template matching. The processor 173 may detect the detection target by inputting the wide-angle image data WD to a classifier that has been trained in advance to detect the detection target based on a predetermined machine learning method, such as a convolutional neural network (CNN). The position information may be, for example, the position coordinates of a bounding box surrounding the detected detection target. The position coordinates of the bounding box may be represented by two-dimensional coordinates in the wide-angle image or by pixel numbers of pixels that constitute the wide-angle image. The processor 173 may also detect each of the worker OP's parts, such as the eyes, face, hands, feet, and torso, and generate position information for each of them. In this case, each of the worker OP's parts, such as the eyes, face, hands, feet, and torso, is included in the detection target. The orientation information of the target object TG, the working device 200, and the mobile device 300 indicates the orientation of the detection target, etc. The posture information of the worker OP refers to information about the posture of each part of the worker OP, such as the eyes, face, hands, feet, and torso.

[0060] The identification information may include luminance information and color information of the detection target. The identification information is not limited to these, and may be information acquired from the wide-angle image data WD and the enlarged image data MD. The identification information may also include information regarding the characteristics of the detection target. The information regarding the characteristics of the target TG may be information regarding at least one of the shape, size, and type of the target TG. The information regarding the characteristics of the detection target may be generated based on the feature amount of the detection target. The feature amount may be, for example, a SIFT (Scale Invariant Feature Transform) feature amount. The identification information may also include partial image data including each detection target. The partial image data is generated, for example, by extracting an area corresponding to a bounding box surrounding each detection target from the wide-angle image. The identification information generates luminance information of the detection target.

[0061] The identification information may include information regarding the time when the wide-angle image data WD and the enlarged image data MD, which represent a wide-angle image including the detection target, were generated. Specifically, the information regarding the time may include information regarding the amount of change in the position, brightness, and color of the detection target. The amount of change regarding time is not limited to these, and may be information acquired from the wide-angle image data WD and the enlarged image data MD. For example, the information regarding the amount of change in the position of the detection target is the difference between the position information of the detection target in the wide-angle image data WD acquired immediately before and the position information of the detection target in the wide-angle image data WD acquired immediately before that. The information regarding the amount of change in the position of the detection target may be information based on the position information of the detection target in three or more pieces of wide-angle image data WD.

[0062] The identification information may include information displayed on the display device 400 that is the detection target. Specifically, the identification information may include information such as an identification number, such as product information, model number, and article number, of the object TG displayed on the display device 400. The processor 173 uses enlarged image data including the display device 400 and enlarged image data including the object TG to obtain the identification number of the object TG being worked on that is displayed on the display device 400, and can also confirm whether the identification numbers are consistent by obtaining the identification number of the object TG that is actually being worked on by the worker OP.

[0063] The processor 173 acquires information about the detection target from the enlarged image data MD. The enlarged image data acquires information about the object, such as item information, position information, or posture information about the worker OP, the target object TG, the working device 200, or the mobile device 300. By using the enlarged image data MD, even if the wide-angle image data WD is not suitable for image analysis because its resolution is insufficient, the enlarged image data MD enables analysis using high-resolution image data by the computing device 170 or an external computing device (not shown). Note that the external computing device is provided separately from the image processing system SYS and has a hardware configuration similar to that of the computing device 170. The processor of the computing device 190 can perform different processes in addition to at least some of the processes performed by the processor 173 of the computing device 170.

[0064] (2) First Example FIG. 3 shows a first example of wide-angle image data and enlarged image data.

[0065] The wide-angle image data WD1 in the first example represents a situation in which a worker OP is working on an object TG1 out of multiple objects TG transported by a conveyor CV. The wide-angle image data WD1 includes a partial image of at least one of the object TG1 and the worker OP. The enlarged image data MD1 in the first example represents an enlarged enlarged target area MA1 of the situation represented in the wide-angle image data WD1, as a first portion. The wide-angle image data WD1 can also be said to represent at least one of the positions, number, postures, and identification information of the objects TG1-TG4, and the state, behavior, and posture of the worker OP.

[0066] The enlargement target area MA1 includes the object TG1. The processor 173 may set the enlargement target area MA1 to include at least a portion of the object TG1 that is closest to the worker OP among the multiple objects TG represented in the wide-angle image data WD1.

[0067] The enlarged image data MD1 represents an enlarged image of the object TG1. The enlarged image data MD1 can be said to represent at least one of the number, posture, position, and identification information of the object TG1. The identification information of the object TG1 may be represented by at least one of characters and symbols (e.g., a one-dimensional barcode or a two-dimensional barcode) on the surface of the object TG1.

[0068] The object recognition area ORA is set to limit the area of ​​the wide-angle image WD1 in which the enlarged image data MD1 is detected. Setting the object recognition area ORA can prevent the processor 173 from falsely detecting the detection target. For example, in FIG. 3 , when the worker OP acquires enlarged image data MD1 of the object TG1 he is working on as the enlarged image area MA1, there are four objects TG1-TG4 represented in the wide-angle image data WD1. The processor 173 may erroneously recognize objects TG2-TG4 other than the object TG1 as the detection target, resulting in the acquisition of enlarged image data MD1 of the objects TG2-TG4. By setting the object recognition area ORA to prevent false recognition, the processor 173 can register the object TG as a detection target in advance, thereby recognizing only the object TG1 included at least partially within the object recognition area ORA as the detection target, and acquire the enlarged image data MD1 of the object TG1. Note that when the conveyor CV is driven so that the object TG1 is no longer included in the object recognition area ORA, acquisition of enlarged image data MD1 for the object TG1 ends. Furthermore, if the conveyor CV is driven so that a different object TG2 is included in at least a portion of the object recognition area ORA, enlarged image data MD1 for the object TG2 can be acquired. Note that the detection target detected by the processor 173 in the object recognition area ORA is not limited to the object TG; if multiple detection targets, such as a portion of the worker OP described below, are registered as detection targets in advance, multiple different detection targets can be detected.

[0069] The number of object recognition areas ORA is not limited to one, and multiple object recognition areas ORA may be set within the wide-angle image data WD1. When multiple object recognition areas ORA are set, it is preferable that the object recognition areas do not overlap with each other in order to prevent the processor 173 from erroneously recognizing the detection target.

[0070] FIG. 4 is a second example of wide-angle image data and enlarged image data.

[0071] The wide-angle image data in the second example represents the same situation as the wide-angle image data WD1 in the first example, and therefore the same reference numerals are used and the description thereof is omitted. The enlarged image data MD2 in the second example represents an enlarged version of the situation represented in the wide-angle image data WD1, with the enlargement target area MA2 being taken as the first part.

[0072] The enlargement target area MA2 includes the face OPf of the worker OP. The wide-angle image data WD1 can be said to represent the state of the face OPf of the worker OP. The processor 173 may set the enlargement target area MA2 so that it includes the face OPf of the worker OP represented in the wide-angle image data WD1.

[0073] The enlarged image data MD2 represents an enlarged view of the face OPf of the worker OP. The processor 173 detects the gaze direction GD of the worker OP from the enlarged image data MD2. Based on the enlarged image data MD2, the processor 173 identifies an object TG2 located in the gaze direction GD of the worker OP as an object located at a point of interest. The point of interest is not limited to one of the multiple objects TG, but may be at least a portion of an object included in the work environment OE, such as the working device 200, the moving device 300, or the conveyor CV. The processor 173 may set a new enlargement target region to include the identified point of interest. The processor 173 may control the imaging device 100 to generate enlarged image data representing the new enlargement target region enlarged as a second portion.

[0074] FIG. 5 is a third example of wide-angle image data and enlarged image data.

[0075] The wide-angle image data in the third example represents the same situation as the wide-angle image data WD1 in the first example, and therefore the same reference numerals are used and the description thereof is omitted. The enlarged image data MD3 in the third example represents an enlarged version of the situation represented in the wide-angle image data WD1, with the enlargement target area MA3 being taken as the first part.

[0076] The enlargement target region MA3 includes the hand OPh of the worker OP. The processor 173 may set the enlargement target region MA3 so as to include the hand OPh of the worker OP represented in the wide-angle image data WD1.

[0077] The enlarged image data MD3 represents an enlarged state of the hand OPh of the worker OP. The processor 173 may generate output data based on the state of the hand OPh of the worker OP, and output the output data to the output device 174. The generation of the output data will be described later.

[0078] (3) Second Example FIG. 6 is a diagram for explaining the imaging of the face OPf of the worker OP using a mirror.

[0079] In the example of FIG. 6 , a worker OP works on an object TG placed on a workbench WB. A mirror M is placed on the workbench WB. The mirror M can also be said to be located around the worker OP. The imaging device 100 is placed above the workbench WB, facing the workbench WB, and generates wide-angle image data WD2 having a wide-angle field of view VFW. The wide-angle image data WD2 represents the situation in which the worker OP works on the object TG placed on the workbench WB. Although not shown, a light source mirror or a light source for illuminating the face OPf of the worker OP may be installed around the mirror M.

[0080] The wide-angle image data WD2 includes an enlargement target area MA4 as a first portion. The imaging device 100 generates enlarged image data MD having an enlarged field of view range VFM and representing the enlargement target area MA4. The enlargement target area MA4 represents at least the face OPf' of the worker OP reflected in the mirror M. The processor 173 may generate output data based on the state of the face OPf' of the worker OP reflected in the mirror M and output the output data to the output device 174. The generation of the output data will be described later.

[0081] 6, an area MA5 to be enlarged (not shown) may be included as the second part. For example, when the face OPf′ of the worker OP is defined as the first part, the processor 173 can analyze the enlarged image data MD of the first part, and acquire enlarged image data as the second part by detecting an object or place in the line of sight of the worker OP.

[0082] (4) Acquisition of enlarged image data MD when there are multiple regions to be enlarged The processor 173 may control the modification device 160 so that an image of the first portion and an image of the second portion are alternately formed on the imaging surface of the second imaging element 151. For example, after an image of the first portion is captured by the second imaging element 151, the processor 173 rotates the reflecting member 161 so that an image of the second portion is formed on the imaging surface of the second imaging element 151. Furthermore, after an image of the second portion is captured by the second imaging element 151, the processor 173 rotates the reflecting member 161 so that an image of the first portion is formed on the imaging surface of the second imaging element 151. This makes it possible to obtain enlarged image data MD of multiple portions (first enlarged image data representing the first portion and second enlarged image data representing the second portion) using a single imaging element.

[0083] The processor 173 controls the second imaging device 150 to generate image data for each predetermined imaging cycle. The time at which the second imaging device 150 generates image data does not have to be the same as the time at which the first imaging device 140 generates image data. For example, the processor 173 may control the second imaging device 150 to generate image data for each second imaging cycle, the second imaging cycle having a length different from that of the first imaging cycle for which the first imaging device 140 generates image data. The processor 173 may also control the second imaging device 150 to generate image data according to a shooting cycle of the same length but starting midway through the first imaging cycle for which the first imaging device 140 generates image data. The image data generated by the second imaging device 150 is enlarged image data MD representing a first portion or enlarged image data MD representing a second portion. The enlarged image data MD includes multiple still image data corresponding to multiple times. The processor 173 acquires the enlarged image data MD generated by the second imaging device 150.

[0084] When the second imaging device 150 generates the first enlarged image data and the second enlarged image data, the processor 173 may determine at least one of the imaging period (first frame rate) when generating the first enlarged image data and the imaging period (second frame rate) when generating the second enlarged image data based on the information of the first part and the information of the second part represented in the wide-angle image data WD.

[0085] For example, the processor 173 compares the priority associated with the first identification information identifying the first portion with the priority associated with the second identification information identifying the second portion, and determines at least one of the first frame rate and the second frame rate so that the imaging period for generating enlarged image data representing the portion with the higher priority is higher. For example, if the priority of the first portion is higher than the priority of the second portion, the processor 173 at least one of determining a higher first frame rate and a lower second frame rate. The priorities may be set in advance according to the importance of the work in the work environment OE. For example, in a work environment OE including multiple objects TG, the priority of an object TG2 that the worker OP is working on may be set higher than the priority of an object TG1 that the worker OP is not working on.

[0086] The processor 173 may perform rotation correction on the enlarged image indicated by the acquired enlarged image data MD. That is, on the imaging surface of the second imaging element 151, the image of the first portion rotates according to the rotation angle of the reflecting member 161, and therefore the image of the first portion included in the enlarged image also rotates. The processor 173 performs rotation correction on the enlarged image so that the rotated image of the first portion is oriented in a predetermined direction. The relationship between the rotation angle for correcting the image of the first portion and the rotation angle of the reflecting member 161 may be stored in advance in the memory 171. The relationship between the rotation angle of the reflecting member 161 and position information of the first portion represented in the wide-angle image data WD may also be stored in advance in the memory 171.

[0087] The processor 173 determines whether the acquired enlarged image data MD was generated after the time required for driving has elapsed since the time when driving of the drive device 163 started. The time required for driving is the time required for the image of the first portion to transition from a state in which an image of the second portion is formed on the imaging surface of the second imaging element 151 to a state in which an image of the second portion is formed on the imaging surface of the second imaging element, or the time required for the image of the second portion to transition from a state in which an image of the second portion is formed on the imaging surface of the second imaging element 151 to a state in which an image of the first portion is formed on the imaging surface of the second imaging element. The time required for driving the drive device 163 may be a fixed value, or may be a value calculated based on the rotation angle of the reflecting member 161 before driving the drive device 163 and the rotation angle of the target reflecting member 161. The fixed value is, for example, 20 ms.

[0088] If the enlarged image data MD was generated before the time required for driving the drive device 163 has elapsed, the processor 173 deletes the acquired enlarged image data MD. If the enlarged image data MD was generated after the time required for driving the drive device 163 has elapsed, the processor 173 stores the acquired enlarged image data MD in the memory 171 in association with the time at which the enlarged image data MD was generated. The processor 173 may also store the enlarged image data MD in the memory 171 in association with wide-angle image data generated at the same time as the time at which the enlarged image data MD was generated or at the most recent time in the past. Enlarged image data MD acquired before the time required for driving the drive device 163 has elapsed is likely to not depict the desired object. Therefore, by the processor 173 storing only enlarged image data MD generated after the time required for driving the drive device 163 has elapsed, only enlarged image data MD depicting the desired object is stored.

[0089] The processor 173 associates identification information with the enlarged image data MD stored in the memory 171. The processor 173 associates first identification information with the enlarged image data MD representing the first portion, and second identification information with the enlarged image data MD representing the second portion. Which portion the enlarged image data MD represents is determined based on whether the processor 173 controlled the modification device 160 based on the position information of the first portion or the position information of the second portion. For example, the processor 173 may associate the first identification information with the enlarged image data MD acquired after controlling the modification device 160 based on the position information of the first portion, and associate the second identification information with the enlarged image data MD acquired after controlling the modification device 160 based on the position information of the second portion.

[0090] The processor 173 may create enlarged image data MD using the wide-angle image data WD. For example, the processor 173 may create the enlarged image data MD using a portion of the wide-angle image data WD that corresponds to the first portion. The processor 173 may create the enlarged image data MD by applying a known pixel interpolation technique to the portion of the wide-angle image data WD that corresponds to the first portion.

[0091] (5) Creation of Moving Images (5-1) Creation of Moving Images Using Enlarged Image Data MD of Multiple Locations FIG. 7 is a diagram for explaining creation of moving images using enlarged image data MD.

[0092] The arithmetic unit 170 acquires first enlarged image data MD4 representing the first portion and second enlarged image data MD5 representing the second portion from the imaging device 100. The first enlarged image data MD4 includes a plurality of first enlarged still image data MD4-1 to MD4-7 generated in time series and corresponding to a plurality of times, respectively. The second enlarged image data MD5 includes a plurality of second enlarged still image data MD5-1 to MD5-7 generated in time series and corresponding to a plurality of times, respectively.

[0093] The memory 171 of the calculation device 170 prestores a first time interval TD1 for setting a plurality of first image selection timings for selecting first enlarged still image data to be used to create the first enlarged video MV1 from the first enlarged image data MD4, and a second time interval TD2 for setting a plurality of second image selection timings for selecting second enlarged still image data to be used to create the second enlarged video MV2 from the second enlarged image data MD5. The processor 173 sets the first image selection timing and the second image selection timing using the first time interval TD1 and the second time interval TD2. In the example of FIG. 7, t １ , t ２ , t ３ are examples of the first image selection timing and the second image selection timing, respectively. In the example of FIG. 7 , the first image selection timing and the second image selection timing are set at the same time using the first time interval TD1 and the second time interval TD2, each of which has the same length, but the setting of the first image selection timing and the second image selection timing is not limited to this. The first time interval TD1 and the second time interval TD2 may have different lengths. Furthermore, the first image selection timing and the second image selection timing may be set with a time lag between them, using the first time interval TD1 and the second time interval TD2, each of which has the same length.

[0094] The processor 173 creates the first enlarged video MV1 using the latest first enlarged image data MD4 at the first image selection timing. ２ , the processor 173 selects the latest first enlarged still image data MD4-3 for creating the first enlarged moving image MV1. ３ The latest second enlarged still image data MD5-6 is selected for creating the second enlarged moving image MV2.

[0095] The processor 173 sequentially performs data processing such as format conversion and compression for creating a moving image on the selected enlarged image data. When performing data processing on multiple enlarged image data within a certain time interval, the processor 173 preferably controls the data processing on each enlarged image data so that the times at which the data processing is performed do not overlap (so that data processing on one enlarged image data is performed at the same time). For example, in the example of FIG. 7, at the first image selection timing t ２ and the first image selection timing t ３ In the time interval between the first image selection timing t １ and the first image selection timing t ２ and data processing DP1-2 for the first enlarged still image data MD4-3 selected for the time interval between the second image selection timing t １ and the second image selection timing t ２ The processor 173 starts the data processing DP1-2 at the first image selection timing t2, and executes the data processing DP2-2 on the second enlarged still image data MD5-3 selected for the time interval between the first image selection timing t ２ to the first image selection timing t ２ and the first image selection timing t ３ By starting the data processing DP2-2 after half the time between the two has elapsed, it is possible to control the times at which the data processing for each enlarged image data is performed so as not to overlap.

[0096] FIG. 8A is a diagram illustrating a first example of image selection using a search range.

[0097] The arithmetic unit 170 acquires enlarged image data MD6 representing the first portion from the image capture device 100. The enlarged image data MD6 includes a plurality of enlarged still image data MD6-1 to MD6-4 that are generated in time series and correspond to a plurality of times, respectively.

[0098] The processor 173 selects enlarged still image data to be used to create the enlarged video MV3 from the enlarged image data MD6 at the image selection timing t １ , t ２ , t３ The image selection timing may be set in the same manner as in the case of creating a moving image using enlarged image data described with reference to FIG.

[0099] The processor 173 creates an enlarged video MV3 using the latest enlarged still image data in a search range that goes back by a search time TD3 from each image selection timing. ３ In order to create the enlarged video MV3, the processor 173 selects the most recent enlarged still image data MD6-3 in the search range SW1, which is located back by search time TD3 from the enlarged still image data MD6-1. While FIG. 7 shows an example of selecting an image from one enlarged image data, the processor 173 may similarly select enlarged still image data to be used to create the enlarged video from each enlarged image data even if there are multiple enlarged image data (e.g., two). In this case, the search time used to determine the search range for each enlarged image data may be the same or different.

[0100] FIG. 8B is a diagram illustrating a second example of image selection using a search range.

[0101] The arithmetic unit 170 acquires enlarged image data MD7 representing the first portion from the imaging device 100. The enlarged image data MD7 includes a plurality of enlarged still image data MD7-1 and MD7-2 that are generated in time series and correspond to a plurality of times, respectively.

[0102] The processor 173 selects enlarged still image data to be used for creating an enlarged video MV4 from the enlarged image data MD7 at an image selection timing t １ , t ２ , t ３ The image selection timing may be set in the same manner as in the case of creating a moving image using enlarged image data described with reference to FIG.

[0103] The processor 173 creates an enlarged video MV4 using the latest enlarged still image data in a search range that goes back by a search time TD4 from each image selection timing. ３In this case, the processor 173 does not include the enlarged still image data in the search range SW2 that is located back by the search time TD4 from the image selection timing t ３ The complementary image data CD is used as the enlarged image data corresponding to the image data. The stored image data may be, for example, a blackout image, a brightening image, or enlarged image data corresponding to the most recent image selection timing. While FIG. 8 shows an example of image selection for one enlarged image data, the processor 173 may similarly select enlarged still image data to be used to create an enlarged video from each enlarged image data even when there are multiple enlarged image data (e.g., two). In this case, the search time used to determine the search range for each enlarged image data may be the same or different.

[0104] FIG. 8C is a diagram illustrating a third example of image selection using a search range.

[0105] The arithmetic unit 170 acquires first enlarged image data MD8 representing the first portion and second enlarged image data MD9 representing the second portion from the imaging device 100. The first enlarged image data MD8 includes a plurality of first enlarged still image data MD8-1 to MD8-3 generated in time series and corresponding to a plurality of times, respectively. The second enlarged image data MD9 includes a plurality of second enlarged still image data MD9-1 to MD9-2 generated in time series and corresponding to a plurality of times, respectively.

[0106] As described above, the imaging device 100 can generate first enlarged image data MD8 representing the first portion and second enlarged image data MD9 representing the second portion by controlling the modification device 160 so that an image of the first portion and an image of the second portion are alternately formed on the imaging surface of the second imaging element 151. In order to generate the second enlarged image data MD9 after generating the first enlarged image data MD8 at a certain time, the imaging device 100 changes the optical path of light incident on the modification device 160 by rotating the reflecting member 161 provided in the modification device 160.

[0107] It is considered that the enlarged image data generated while the modification device 160 is changing the optical path of the light does not appropriately represent either the first portion or the second portion. Therefore, the processor 173 may select enlarged still image data for generating an enlarged video from the enlarged image data, excluding enlarged image data corresponding to the time when the modification device 160 is changing the optical path of the light. It can also be said that the processor 173 sets the range of the enlarged image data, excluding the enlarged image data corresponding to the time when the modification device 160 is changing the optical path of the light, as a search range, and selects enlarged still image data from the search range.

[0108] In the example of FIG. 8C, the processor 173 selects the first image at the first image selection timing t ２ Corresponding to this, the processor 173 selects first enlarged still image data MD8-2 from the first enlarged image data MD8 and selects second enlarged still image data MD9-1 from the second enlarged image data MD9. If there are multiple pieces of target enlarged still image data at a certain image selection timing, the processor 173 may select the most recent enlarged still image data as described with reference to FIG. 7. Alternatively, the processor 173 may select enlarged still image data from a search range that is set by the search time prior to the image selection timing, as described with reference to FIG. 8(A). Alternatively, the processor 173 may use complementary image data if the search range does not include enlarged still image data, as described with reference to FIG. 8(B).

[0109] Processor 173 may create a wide-angle video using the method described above in Figures 7 and 8. By using the method described above, it is possible to reduce the amount of wide-angle image data required to create a wide-angle video compared to creating a wide-angle video using all of the wide-angle image data, thereby reducing the amount of data in the wide-angle video.

[0110] (5-2) The video division processor 173 may create an enlarged video using the enlarged image data MD. For example, the processor 173 creates and outputs an enlarged video in a predetermined video format, such as MP4 or MPEG, so that at least some of the still image data included in the enlarged image data MD, each corresponding to a different time, are displayed in the order in which they were generated. The processor 173 may also create and output an enlarged still image in a predetermined still image format, such as JPEG or PNG, from any of the still image data included in the enlarged image data MD. In this case, the processor 173 may store the enlarged image data MD acquired from the imaging device 100 in the memory 171 and create an enlarged video using the stored enlarged image data MD. The processor 173 may also create two enlarged videos for each of the first and second parts. The number of enlarged images is not limited to one or two.

[0111] The memory 171 of the calculation device 170 prestores a first time interval TD1 for setting a plurality of first image selection timings for selecting first enlarged still image data to be used to create the first enlarged video MV1 from the first enlarged image data MD4, and a second time interval TD2 for setting a plurality of second image selection timings for selecting second enlarged still image data to be used to create the second enlarged video MV2 from the second enlarged image data MD5. The processor 173 sets the first image selection timing and the second image selection timing using the first time interval TD1 and the second time interval TD2. In the example of FIG. 7, t １ , t ２ , t ３are examples of the first image selection timing and the second image selection timing, respectively. In the example of FIG. 7 , the first image selection timing and the second image selection timing are set at the same time using the first time interval TD1 and the second time interval TD2, each of which has the same length, but the setting of the first image selection timing and the second image selection timing is not limited to this. The first time interval TD1 and the second time interval TD2 may have different lengths. Furthermore, the first image selection timing and the second image selection timing may be set with a time lag between them, using the first time interval TD1 and the second time interval TD2, each of which has the same length.

[0112] The processor 173 creates the first enlarged video MV1 using the latest first enlarged image data MD4 at the first image selection timing. ２ , the processor 173 selects the latest first enlarged still image data MD4-3 for creating the first enlarged moving image MV1. ３ The latest second enlarged still image data MD5-6 is selected for creating the second enlarged moving image MV2.

[0113] The processor 173 may detect at least one of first trigger information that sets the start timing of the enlarged video and second trigger information that sets the end timing of the enlarged video from the wide-angle image data WD. In this case, the processor 173 generates the enlarged video using the enlarged image data MD associated with the time included in the time interval from the start timing to the end timing.

[0114] The trigger information may be, for example, position information or posture information of the worker OP, the target object TG, or the working device 200 in the wide-angle image data WD, or the detection target described above may be the trigger information.

[0115] A specific example is shown in FIG. 3 . Reference lines RL1 and RL2 are provided in the wide-angle image data WD. The reference lines RL1 and RL2 may be displayed on the display screen SCR. As shown in FIG. 3 , when the conveyor CV transports objects from left to right, the timing when at least a portion of the object TG1 passes through the reference line RL1 can be used as first trigger information. Furthermore, the timing when at least a portion of the object TG1 passes through the reference line RL2 can be used as second trigger information. This allows the processor 173 to create a video of the work performed by the worker OP on the object TG1. Furthermore, the wide-angle image data WD while the detected object is included in the object recognition area ORA can be used as a wide-angle video. Setting of trigger information based on position information is not limited to this specific example and can be set arbitrarily.

[0116] A specific example is shown in FIG. 10 . FIG. 10 is a diagram illustrating the movement of a worker within a predetermined area. A work environment OE is shown, with a workbench WB and shelves R1-R4 arranged thereon. A worker OP works on an object TG arranged on the workbench WB. During the work, the worker OP moves along a movement path MP indicated by a dotted line between an area A0 near the workbench WB and areas A1-A3 near each of the shelves R1-R3. The work performed by the worker OP may be, for example, loading items stored on the shelves R1-R3 onto the object TG. FIG. 10 can also be considered a top view of the work environment OE. The area of ​​the work environment OE enclosed by a frame WD3 is acquired as wide-angle image data WD. The processor 173 uses the wide-angle image data WD3 to learn the movements of the worker OP and the object TG in the normal area and the abnormal area. The normal area is an area in the wide-angle image data WD that includes the movement path MP through which the worker OP passes when performing work correctly. The normal area can also be said to be an area where work is performed appropriately on the target object TG by at least one of the worker OP and the work device 200. The normal area can also be expressed as a reference environment or reference data. The variable area refers to an area that is not used in normal work by the worker OP. The variable area may be an area different from the normal area.

[0117] The processor 173 can set the normal area and the change area based on the position information of the wide-angle image data MD of the detection target. For example, the processor 173 can define the normal area as a range within a predetermined threshold distance from each point on the movement path MP. The processor 173 can detect whether the worker OP is in the normal area based on the wide-angle image data WD. The processor 173 can determine the timing at which it is determined that the worker OP is in a change area that is outside the predetermined threshold from the movement path MP as first trigger information. Alternatively, the processor 173 can determine the timing at which it is determined that the worker OP has returned to normal work by staying in the normal area for a certain period of time after the first trigger as second trigger information. This allows for the creation of a video of abnormal work.

[0118] Another specific example of the setting of the normal region and the changed region by the processor 173 can be defined using a heat map HM shown in FIG. 14, which will be described later. After learning, the processor 173 may use the timing when the worker OP or the object TG enters the changed region or the timing when it is detected that the worker OP or the object TG has stayed in the changed region for a certain period of time or more as the first trigger information based on the wide-angle image data WD3. Alternatively, the processor 173 may use the timing when it is determined that the worker OP or the object TG has returned to normal work by staying in the normal region for a certain period of time after the first trigger as the second trigger information. This makes it possible to create a video of abnormal work.

[0119] The processor 173 may set all areas used while work is being performed on the target object TG as the normal area. The processor 173 may also set normal areas by dividing work on shelves R1-R3 by shelf. For example, when the worker OP is performing work on shelf R1, the processor 173 may set areas related to shelves R2 and R3 as not included in the normal area. On the other hand, when the worker OP is performing work on shelf R2, the processor 173 may set areas related to shelves R1 and R3 as not included in the normal area.

[0120] The processor 173 may generate a wide-angle video using wide-angle image data WD associated with a time included in a time interval set based on trigger information detected from the wide-angle image data WD.

[0121] The processor 173 may detect trigger information in response to a change in the detection target (e.g., the posture of the target TG and the position of the worker OP or a predetermined part of the target TG) represented in the wide-angle image data WD. The processor 173 can detect trigger information based on a change in identification information detected from each of a plurality of still image data items corresponding to a plurality of times and included in the wide-angle image data WD.

[0122] The processor 173 may also detect trigger information by inputting the wide-angle image data WD to a classification device that has been trained in advance by machine learning. The classification device may be configured, for example, by including a neural network, inputting a large amount of training data into the neural network, and performing machine learning according to a known learning method such as backpropagation. The training data may include, for example, a large number of still images representing various situations that occur in the work environment OE, and information indicating whether the situation depicted in each still image corresponds to the start or end timing of the enlarged video.

[0123] 7, the processor 173 may divide the enlarged video MV based on the trigger information after creating the first enlarged video MV1 and the second enlarged video MV2 shown in Fig. 7. Alternatively, the processor 173 may create the enlarged video MV based on the trigger information before creating the first enlarged video MV1 and the second enlarged video MV2 shown in Fig. 7.

[0124] The wide-angle video and the enlarged video are stored in memory 171. The wide-angle video and the enlarged video may also be output from output device 174 and stored in an external storage device (not shown). Storing the wide-angle video and the enlarged video in memory 171 or the external storage device enables the stored videos to be played back and analyzed in the event of an abnormality. The external storage device is a storage device provided separately from the image processing system SYS.

[0125] In addition, the memory 171 or the external storage device may overwrite and store new wide-angle videos and enlarged videos when the capacity of the wide-angle videos and enlarged videos exceeds a certain capacity, or when the time spent storing the wide-angle videos and enlarged videos has passed a certain period of time.

[0126] The processor 173 may store the wide-angle video WV and the enlarged video MV as independent videos in the memory 171 or an external storage device. Alternatively, the processor 173 may create a data file in which the wide-angle video WV and the enlarged video MV are displayed on a single display screen as shown in FIG. 9 and store the data file in the memory 171 or an external storage device.

[0127] It should be noted that the creation of the wide-angle video and the enlarged video using the trigger information does not have to be performed by the processor 173. An external computing device may create at least one of the wide-angle video and the enlarged video using the wide-angle image data WD and the enlarged image data MD output from the output device 174.

[0128] (6) The wide-angle image data WD and enlarged image data MD output processor 173 generates output data based on at least one of the wide-angle image data WD and enlarged image data MD, and outputs the data to the output device 174 .

[0129] The output data may include, for example, as shown in FIG. 1, at least one of a display signal for displaying a predetermined image on the display of the display device 174-1, an audio signal for generating a predetermined sound from the speaker 174-2, a light emitting signal for causing the indicator light 174-3 to emit light in a predetermined pattern, a display signal for displaying a predetermined image on the wearable terminal, a control signal for causing the wearable terminal to generate sound or vibration, and a display signal for displaying a predetermined image on the smart glasses.

[0130] (6-1) Output Example 1 Figure 9(A) is an example of a screen showing both wide-angle video and enlarged video, Figure 9(B) is an example of wide-angle image data showing an object being transported, and Figure 9(c) is an example of enlarged image data showing an object being transported.

[0131] The display signal output by processor 173 to display device 174-1 as output data causes the display of display device 174-1 to display, for example, the display screen SCR shown in FIG. 9A. A wide-angle video WV is arranged on the left side of display screen SCR, and enlarged videos MV7, MV8, and MV9 are arranged, in order from the top left, to the right of the wide-angle video WV. In this way, arithmetic device 170 can output a display signal for displaying a screen showing both the wide-angle video and the enlarged video.

[0132] On the display screen SCR, a corresponding still image may be displayed instead of at least one of the wide-angle video WV and the enlarged videos MV7-MV9.

[0133] (6-2) Output Example 2 Figures 9B and 9C are diagrams for explaining a method for displaying a detection target on the display screen SCR when the position information of the detection target changes within a predetermined threshold. １ -t ４ Wide-angle image data WD representing the situation in which the object TG is conveyed on the conveyor CV in each of the ｔ１ -WD ｔ４ Wide-angle image data WD ｔ１ -WD ｔ４ In the image data WD shown in FIG. 9B, bounding boxes BB1 and BB2 are displayed, which surround the object TG to be detected by the processor 173. ｔ１ -WD ｔ４ may or may not be displayed on the display screen SCR. １ -t ４ In the enlarged video MV representing the detection target generated corresponding to the time t １ -t ４ The corresponding expanded video data MV ｔ１ -MV ｔ４ Represents the expanded video data MV ｔ１ -MV ｔ４ Each of the above may be referred to as enlarged image data. The enlarged moving image MV may be displayed on the display screen SCR.

[0134] When acquiring enlarged image data MD of the object TG to be detected, the processor 173 can acquire the image using the bounding box BB as a reference. At this time, the center of gravity of the bounding box BB may shift due to slight movements of the object TG or detection errors by the processor 173. In such cases, the position of the object TG is displayed slightly shifted in the enlarged image data MD. As a result, an enlarged video MV using the acquired enlarged image data MD is generated in which the object TG appears to move slightly up, down, left, and right on the display screen SCR. Using the method described below, it is possible to create an enlarged video MV with minimal blur. The method described below is preferably used when creating an enlarged video MV of the object TG in a situation where no work is being performed by the worker OP or the work device 200.

[0135] The processor 173 １ Wide-angle image data WD at (first timing) ｔ１ The target object TG is detected as a detection target using １ Position information of the bounding box BB1 of the object TG at time t １ The position information of the object TG at time t is the reference position. In this description, the center of gravity position of the object TG is the reference position. The position information of the bounding box BB is the position information of a predetermined part such as the center of gravity of the bounding box BB in the wide-angle image data WD or the imaging area of ​​the first imaging element 141. The position information of the object TG is the position information of a predetermined part such as the center of gravity of the object TG in the wide-angle image data WD or the imaging area of ​​the first imaging element 141. １ The display screen SCR at time t １ Enlarged video data MV of the object TG in ｔ１ is displayed.

[0136] Next, the processor 173 calculates the time t ２ Wide-angle image data WD at (second timing) ｔ２ The processor 173 detects the target object TG at time t ２The center of gravity position of the object TG at time t is determined to be within a range of a predetermined threshold value from the reference position. ２ The display screen SCR at time t １ Enlarged video data MV of the object TG in ｔ１ continues to be displayed.

[0137] The predetermined threshold indicates, for example, the length from the reference position. As a specific example, the predetermined threshold may be a specified number of pixels from the reference position in the imaging area of ​​the first imaging element 141, may be determined depending on the size of the target object TG, or may be set arbitrarily. The length from the reference position may be calculated using a known method such as a moving average.

[0138] Next, the processor 173 calculates the time t ３ Wide-angle image data WD at (third timing) ｔ３ The processor 173 detects the target object TG at time t ３ If it is determined that the center of gravity position of the object TG at time t １ Enlarged video data MV of the object TG in ｔ１ The display of time t ３ The display screen SCR in ｔ１ Instead of the time t ３ When it is determined for the first time that the center of gravity of the object TG is farther from the reference position than the range of a predetermined threshold value, the enlarged image data MD to be used for the enlarged video MV is １ Enlarged video data MV of the object TG in ｔ１ may still be displayed.

[0139] Furthermore, the processor 173 calculates the time t ４ At (fourth timing), wide-angle image data WDｔ４ The processor 173 detects the target object TG at time t ４ The center of gravity of the object TG at time t ３ The processor 173 determines that the position exceeds the predetermined threshold range from the reference position in the same manner as in the case of the time t １ Enlarged video data MV of the object TG in ｔ１ Instead of displaying the image data, the complementary image data CD such as a darkened image or a brightened image may be displayed.

[0140] (6-3) Output Example 3 As shown in FIG. 1 , the processor 173 may prompt the worker OP to change the work being done on the object TG by outputting output data to the output device 174. For example, the processor 173 generates an audio signal for generating a sound that prompts the worker OP to change the work being done on the object TG, and outputs the signal as output data to the speaker 174-2. The processor 173 also generates a light-emitting signal for causing the indicator light 174-3 to emit light in a pattern that prompts the worker OP to change the work being done on the object TG, and outputs the signal as output data to the indicator light 174-3. Such output data can also be considered a notification that prompts the worker OP to change the work being done on the object TG.

[0141] The processor 173 may output the output data to other devices, such as the working device 200 and the mobile device 300, via the interface 172. For example, the processor 173 may generate a work control signal for controlling the work performed by the working device 200 on the target object TG based on at least one of the wide-angle image data WD and the enlarged image data MD, and output the signal to the working device 200. The work control signal is preferably generated based on the enlarged image data MD that enlarges and represents the first portion. The processor 173 may also generate a movement control signal related to the movement of the working device 200 or the mobile device 300, and output the movement control signal to the working device 200 or the mobile device 300. The movement control signal is preferably generated based on the enlarged image data MD that enlarges and represents the first portion. The movement control signal related to the working device 200 may be, for example, a control signal output to at least one working device 200 to prevent a collision between adjacent working devices 200-1 and 200-2. If the working device 200 itself has a movable member that can move, the control signal may be related to the movement of the working device 200. The movement control signal for the moving device 300 may be, for example, a control signal output to move within the work environment OE, or a control signal output to the moving device 300 to prevent contact with a worker OP or conveyor CV within the work environment OE.

[0142] The processor 173 may determine whether the enlarged image data MD satisfies a predetermined abnormal condition. In this case, the processor 173 notifies the worker OP to change the work performed on the object TG only if the enlarged image data MD satisfies the abnormal condition. The processor 173 also outputs a work control signal to the work device 200 based on the determination result of whether the enlarged image data MD satisfies the abnormal condition. The abnormal condition may be, for example, a percentage of the situation represented in the enlarged image data MD that does not correspond to the situation represented in the reference environment previously stored in the memory 171 exceeding a predetermined value.

[0143] The processor 173 may determine the quality of the work results performed by the worker OP or the work device 200 based on at least one of the wide-angle image data WD and the enlarged image data MD. In this case, the processor 173 notifies the worker OP of the quality determination result in accordance with the quality determination result of the work result. The processor 173 also outputs a work control signal to the work device 200 in accordance with the quality determination result of the work result. It is preferable that the quality of the work result be determined based on the enlarged image data MD representing the work result.

[0144] The processor 173 may generate a still image representing the movement of at least one of the object TG and the worker OP represented in the wide-angle image data WD. The imaging device 100 generates wide-angle image data WD representing a situation in which the worker OP performs work on the object TG while at least one of the object TG and the worker OP moves within a predetermined area.

[0145] FIG. 10 is a diagram illustrating the movement of a worker within a predetermined area.

[0146] The work environment OE includes a workbench WB and shelves R1-R4. A worker OP performs work on an object TG placed on the workbench WB. During work, the worker OP moves along a movement path MP indicated by a dotted line between an area A0 near the workbench WB and each of areas A1-A3 near the shelves R1-R3. The work performed by the worker OP may, for example, be loading items stored on the shelves R1-R3 onto the object TG. FIG. 10 can also be considered a top view of the work environment OE. Note that after completing a series of work (process) on one object TG, the worker OP may perform the same process on a new object. In this case, the movement path MP in the process for the one object TG and the movement path in the process for the new object are not necessarily the same.

[0147] The image capturing device 100 is installed above the work environment OE and captures images facing downward. In Fig. 10, the area indicated by the solid line and labeled "WD3" corresponds to the portion of the work environment OE represented in the wide-angle image data WD3 generated by the image capturing device 100.

[0148] The position of the worker OP within the area represented by the wide-angle image data WD3 can be expressed as coordinates in a coordinate system having an X axis and a Y axis that intersect at a reference point O. For example, the position of the worker OP is expressed as (1000, 500) within area A0, (500, 1500) within area A1, (1000, 1500) within area A2, and (1500, 1200) within area A3. The reference position of the position of the worker OP is set to any position, such as the position of the worker OP's head or hands.

[0149] (6-4) Output Example 4 FIG. 11A is a first example of a still image showing the movement of the worker OP.

[0150] The still image SP1 is a two-dimensional graph in which the horizontal axis represents time and the vertical axis represents the X-coordinate or Y-coordinate value of the position of the worker OP in the wide-angle image data WD3 at each time. The data series DSX1 represented in the still image SP1 corresponds to the X-coordinate of the position of the worker OP in the wide-angle image data WD3, and the data series DSX2 corresponds to the Y-coordinate of the position of the worker OP in the wide-angle image data WD3. The still image SP1 can also be considered a movement trajectory image that represents the change in position (movement) of the worker OP within a predetermined period of time. The predetermined period is, for example, from the start to the end of work on one object TG. The processor 173 may output the still image SP1 as output data to the display device 174-1.

[0151] FIG. 11B is a second example of a still image showing the movement of the worker OP.

[0152] Similar to the still image SP1, the still image SP2 is a two-dimensional graph with the horizontal axis representing time and the vertical axis representing the value of the X coordinate or Y coordinate in the wide-angle image data WD3 of the position of the operator OP at each time. In the still image SP2, in addition to the still image SP1, a data range DRX having a value of the X coordinate corresponding to the region A0 (for example, 950 < x < 1050) and a data range DRY having a value of the Y coordinate corresponding to the region A0 (for example, 450 < y < 550) are shown. At the time when the value of the data series DSX2 is within the data range DRX and the value of the data series DSY2 is within the data range DRY, it can be seen that the operator OP is located within the region A0. In FIG. 11(B), such times are shown as work WPa1, work WPa2, work WPa3, and work WPa4.

[0153] Similarly, from the still image SP2, the times when the operator OP is located within the regions A1 - A3 can be specified. In FIG. 11(B), such times are shown as work WPb, work WPc, and work WPd, respectively.

[0154] At the times between each work shown in this way, at least one of the value of the X coordinate and the value of the Y coordinate of the position of the operator OP is changing. This indicates that the operator OP is moving. In FIG. 11(B), such times are shown as moving times TT1 - TT6, respectively. The moving time can also be said to be one of the works. The moving time can also be said to be an idle time.

[0155] The still image SP2 can also be said to be a moving trajectory image representing the change (movement) of the position of the operator OP within a predetermined period. The processor 173 may output the still image SP2 as output data to the display device 174 - 1.

[0156] FIG. 11(C) is a third example of a still image representing the movement of the operator OP.

[0157] As described with reference to FIG. 11(B), each work is performed over a time period of a predetermined length. When the operator OP performs steps regarding a plurality of objects TG, the length of time used for each work may vary respectively. The processor 173 may aggregate the lengths of time used for each work in each step regarding the plurality of objects TG.

[0158] Still image SP3 is a two-dimensional graph in which the horizontal axis represents the time from a reference time or the order of tasks, and the vertical axis represents the length (duration) of each task performed during that time. The reference time is the reference time for each task related to multiple objects TG, such as the start time of the first task. The duration is the period during which a task remains within a specified range (the change in position over time is less than the movement threshold), which can also be referred to as the duration of stay. Still image SP3 is a box-and-whisker plot that shows the distribution of duration of stay for each task, with boxes indicating the first and third quartiles and whiskers indicating the minimum and maximum values. Still image SP3 can also be considered a duration of stay image that shows the distribution of duration of stay for each task. Note that the boxes and whiskers may represent other statistical values.

[0159] Although still images SP1-SP3 shown in FIGS. 11A-11C represent the movement of a worker within a predetermined area, the still images may also represent the movement of an object TG. For example, the still images may represent the movement of each of multiple objects TG transported by a conveyor CV. A predetermined number of the multiple objects TG may be contained in multiple storage boxes. That is, the imaging device 100 may generate an image representing a work environment OE in which multiple storage boxes, each containing multiple objects TG, are transported by a conveyor CV. In this case, the wide-angle image data WD may represent the situation in which the multiple storage boxes are transported by the conveyor CV, and the enlarged image data MD may represent at least one object TG contained in one of the multiple storage boxes.

[0160] FIG. 12 is an example of an image of stay periods by process.

[0161] As described with reference to FIG. 11C , when a worker OP performs processes on multiple objects TG, the length of each task may vary. Still image SP4 represents the length of each task in each process (processes C1-C5) for multiple objects TG. Still image SP4 can also be considered a process-specific stay duration image, representing the duration of each task included in each process. Still image SP4 also includes an average process Ave, where the length of each task is the average length of each task. Still image SP4 also includes an average process duration DCAve, which indicates the average process length; an upper limit process duration DCU, which indicates the upper limit of a standard process length; and a lower limit process duration DCL, which indicates the lower limit of the standard process length. The upper limit process duration DCU and the lower limit process duration DCL may be set, for example, to the average process duration DCAve ± the variance of the process duration × 3.

[0162] The processor 173 can determine that a process whose period length falls outside the range of the upper limit process period length DCU and the lower limit process period length DCL is a process in which an abnormality has occurred. In the example of Figure 12, the process period length of process C5 is shorter than the lower limit process period length DCL. Therefore, the processor 173 can determine that an abnormality has occurred in process C5. The upper limit process period length DCU and the lower limit process period length DCL can also be considered examples of abnormal conditions.

[0163] FIG. 13 shows an example of detailed image information.

[0164] The processor 173 may further generate detailed image information for each task included in the process, which represents, in chronological order, information detected from the enlarged image data MD during the duration of stay at that task.

[0165] 13 includes the still image SP3 shown in FIG. 11C. On the display device 174-1 on which the still image SP3 is displayed, the user focuses on one of the tasks (e.g., travel time TT6) and selects it by, for example, using a pointing device such as a mouse or by touching a touch panel superimposed on the display. At this time, the processor 173 further generates detailed image information that chronologically represents information detected from the enlarged image data MD during the duration of the task.

[0166] The detailed image information DP1 is a two-dimensional graph in which the horizontal axis represents time and the vertical axis represents the X-coordinate or Y-coordinate value of the position of the worker OP in the enlarged image data MD at each time. The data series DSX3 represented in the detailed image information DP1 corresponds to the X-coordinate of the position of the worker OP in the enlarged image data MD, and the data series DSX4 corresponds to the Y-coordinate of the position of the worker OP in the enlarged image data MD. The detailed image information DP1 can also be said to be a movement trajectory image that represents the change in position (movement) of the worker OP during the period of the target work.

[0167] The detailed image information DP2 is a two-dimensional graph in which the horizontal axis represents time and the vertical axis represents the average brightness of the enlarged image data MD at each time. The vertical axis of the detailed image information DP2 may represent the maximum brightness of the enlarged image data MD at each time, or the intensity of each RGB color.

[0168] The processor 173 generates such detailed image information and outputs it to the display device 174-1, allowing the user to obtain detailed information about the task of interest. For example, the user can use such detailed image information to analyze the cause of a task that takes longer than the standard amount of time in a certain process.

[0169] (6-5) Output Example 5 FIG. 14 is an example of a heat map.

[0170] The work environment OE shown in Figure 14 is similar to the work environment OE described with reference to Figure 10, and therefore a detailed description thereof will be omitted. The processor 173 calculates the cumulative stay time of the worker OP for each coordinate in a coordinate system having an X axis and a Y axis intersecting at a reference point O in the area represented by the wide-angle image data WD3, based on the position of the worker OP at each time. The processor 173 may create a heat map HM so that coordinates with longer cumulative stay times are displayed darker. The heat map can be considered a still image representing the movement of the worker OP.

[0171] The processor 173 generates such a heat map HM and outputs it as a still image to the display device 174-1, allowing the user to understand the stay time of the worker OP at each position in the process. For example, the user can use such a heat map HM to analyze the arrangement of flow lines that should be noted in order to improve the efficiency of the process. Furthermore, if the position of the worker OP is outside the area of ​​long cumulative stay time represented in the heat map HM, the processor 173 can determine that the worker OP is in an abnormal state.

[0172] The arithmetic unit 170 including the processor 173 that generates a still image based on the wide-angle image data WD in this way can also be called an image processing device.

[0173] The computing device 170 may not include the memory 171. The computing device 170 may be housed in the housing 101. A part of the configuration of the computing device 170 may be housed in the housing 101.

[0174] As described above, the output device 174 includes any one of the display device 174-1, the speaker 174-2, and the indicator light 174-3. The display device 174-1 is a device that displays images, such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display device 174-1 displays images based on a display signal output from the arithmetic device 170. The display device 174-1 may be integrated with the arithmetic device 170. The speaker 174-2 is a device that generates sound. The speaker 174-2 is installed in the work environment OE and generates sound based on the audio signal output from the arithmetic device 170 so that it can be heard by workers working in the work environment OE. The indicator light 174-3 is a device that has a light-emitting unit that emits light in a predetermined pattern (for example, a green light that lights up, a red light that flashes, etc.). The indicator light 174-3 is installed in the work environment OE, and emits light in a predetermined pattern based on a light emission signal output from the computing device 170 so that it can be seen by a worker working in the work environment OE.

[0175] In addition, the output of output data to the output device 174 based on at least one of the wide-angle image data WD and the enlarged image data MD, and the output of work control signals and movement control signals to at least one of the work device 20 and the movement device 300 do not need to be performed by the processor 173, and the above-mentioned control signals may be output by an external calculation device provided separately from the image processing system SYS to which at least one of the wide-angle image data WD and the enlarged image data MD is output from the calculation device 170.

[0176] [Modification] The image processing system of the modification differs from the image processing system SYS in that it includes an imaging device having a variable magnification optical system that forms an image of incident light and is capable of changing the focal length, and an image sensor that generates image data in accordance with the image formed by the variable magnification optical system, instead of the imaging device 100. Parts of the configuration of the image processing system of the first modification that are similar to those of the image processing system SYS are designated by the same reference numerals, and detailed description thereof will be omitted.

[0177] The processor 173 of the arithmetic unit 170 controls the image capturing device 100 so that the focal length of the variable magnification optical system becomes the first focal length. The processor 173 acquires, as wide-angle image data WD, image data generated by the image capturing element in accordance with an image formed when the focal length of the variable magnification optical system becomes the first focal length.

[0178] The processor 173 controls the imaging device 100 so that the focal length of the variable magnification optical system becomes a second focal length that is longer than the first focal length. The processor 173 acquires, as enlarged image data MD, image data generated by the imaging element in accordance with an image formed when the focal length of the variable magnification optical system becomes the second focal length. The processor 173 can also acquire, as enlarged image data MD, image data generated by cropping the wide-angle image data WD.

[0179] It should be understood by those skilled in the art that various changes, substitutions, and modifications can be made to the above-described embodiments within the scope of the present invention. For example, the above-described processes may be performed in a different order within the scope of the present invention. The above-described various embodiments and modifications may be combined as appropriate.

Claims

an imaging device that generates wide-angle image data that represents at least a situation in which a worker is performing work on an object; a computing device that acquires enlarged image data that enlarges and represents a first portion included in an imaging range of the imaging device corresponding to the wide-angle image data, based on the wide-angle image data and reference data related to movements during the work of a detection target including at least one of the worker and the object; the enlarged image data represents at least a portion of the detection target; Image processing system. 　 The image processing system according to claim 1 , wherein the calculation device acquires the enlarged image data that represents an enlarged second portion that is included in the imaging range of the imaging device corresponding to the wide-angle image data and that is different from the first portion. 　 At least one of the first portion and the second portion includes the object; The enlarged image data represents at least one of the number, the posture, and the identification information of the objects. The image processing system according to claim 2 . 　 the wide-angle image data represents a plurality of the objects; At least one of the first portion and the second portion is at least a part of an object that is closest to the worker among the plurality of objects.

4. The image processing system according to claim 2. 　 At least one of the first portion and the second portion includes at least one of the worker's eyes, face, hands, feet, and torso; The enlarged image data represents at least one of the worker's eye direction, face direction, hand position, foot position, and posture. An image processing system according to any one of claims 2 to 4. 　 an imaging device that generates wide-angle image data that represents at least a situation in which a worker is performing work on an object; a calculation device that acquires, based on the wide-angle image data, enlarged image data that enlarges and represents a first portion included in an imaging range of the imaging device corresponding to the wide-angle image data, the wide-angle image data represents a face of the worker; the enlarged image data represents an enlarged image of at least a part of the face of the worker as the first part; the computing device identifies an object in the line of sight of the worker based on the enlarged image data. Image processing system. 　 The image processing system according to claim 6 , wherein the calculation device acquires the enlarged image data representing the face of the worker reflected in a mirror disposed around the worker as the first part. 　 The image processing system according to claim 6 or 7, wherein the calculation device acquires enlarged image data that enlarges and represents a second portion that is included in the imaging range of the imaging device corresponding to the wide-angle image data and is different from the first portion. 　 The image processing system according to claim 8 , wherein the computing device acquires the enlarged image data representing at least one of a hand, a foot, and a torso of the worker as the second portion. 　 The image processing system according to any one of claims 1 to 9, wherein the arithmetic device executes a notification to the worker to prompt the worker to change the work on the object based on the enlarged image data. 　 The image processing system according to any one of claims 1 to 10, wherein the arithmetic device executes a notification to the worker to prompt the worker to change the work being performed on the object based on the wide-angle image data. 　 an imaging device that generates wide-angle image data that represents at least a situation in which at least one of a worker and a work device is performing work on an object; acquiring, based on the wide-angle image data, enlarged image data representing an enlarged first portion included in an imaging range of the imaging device corresponding to the wide-angle image data; a computing device that executes at least one of the following: a notification to the worker prompting the worker to change the work on the object based on the enlarged image data; and an output of a work control signal to the work device that controls the work on the object; An image processing system comprising: 　 The image processing system according to claim 12 , wherein the calculation device acquires the enlarged image data based on the wide-angle image data and reference data relating to the movement during the work. 　 The image processing system according to claim 13 , wherein the reference data is data for causing the arithmetic unit to learn the movements of at least one of the worker and the work device during the work. 　 The image processing system of claim 13 or 14, wherein the calculation device uses a normal area in the wide-angle image data where work is appropriately performed on the object by at least one of the worker and the work device as the reference data, and acquires the enlarged image data when the first part is included in a changed area that is an area different from the normal area. 　 The image processing system according to any one of claims 12 to 15, wherein the calculation device executes a notification to the worker to prompt the worker to change the work being done on the object when the enlarged image data satisfies a predetermined abnormal condition. 　 The image processing system according to any one of claims 12 to 16, wherein the calculation device outputs the work control signal relating to the work performed by the work device on the object when the enlarged image data satisfies a predetermined abnormal condition. 　 The image processing system according to any one of claims 12 to 17, wherein the calculation device outputs a movement control signal regarding the movement of a moving device that is capable of moving around the object and is different from the working device, based on the enlarged image data. 　 The image processing system of claim 18 , wherein the computing device obtains the enlarged image data representing the mobile device, and outputs the movement control signal relating to the movement of the mobile device based on the enlarged image data. 　 The image processing system according to any one of claims 12 to 19, wherein the calculation device acquires the enlarged image data representing the work results performed by the worker or the work device, and determines whether the work results are good or bad based on the enlarged image data.

21. The image processing system of claim 20, wherein the arithmetic device performs at least one of notifying the worker of the result of the pass / fail judgment of the work result and outputting a work control signal to the work device to control the work device to perform the work, depending on the result of the pass / fail judgment of the work result. 　 The image processing system of any one of claims 1 to 21, wherein the arithmetic device generates at least one of a still image and a video based on the wide-angle image data, and generates at least one of a still image and a video based on the enlarged image data. 　 The image processing system of claim 22, wherein the arithmetic device creates a wide-angle video using the wide-angle image data including a plurality of still images corresponding to a plurality of times, and creates an enlarged video using the enlarged image data including a plurality of still images corresponding to a plurality of times. 　 an imaging device that generates wide-angle image data representing a situation including an object in a time series; acquiring, based on the wide-angle image data, enlarged image data representing an enlarged first portion included in an imaging range of the imaging device corresponding to the wide-angle image data; a computing device that creates a moving image including at least one of a wide-angle moving image using the wide-angle image data and an enlarged moving image using the enlarged image data, The computing device detecting, from the wide-angle image data, first trigger information that sets a start timing of the moving image among trigger information that sets at least one of a start timing and an end timing of the moving image; detecting second trigger information from the wide-angle image data after the start timing, the second trigger information setting an end timing of the video; generating the moving image using at least one of the wide-angle image data and the enlarged image data from the start timing to the end timing; Image processing system.

25. The image processing system according to claim 24, wherein the arithmetic unit detects the trigger information in response to a change in at least one of the posture of the object and the position of a predetermined part of the object depicted in the wide-angle image data. 　 The image processing system according to claim 24 or 25, wherein the arithmetic unit detects the trigger information by inputting the wide-angle image data into a classification unit that has been trained in advance by machine learning. 　 the enlarged image data includes first enlarged image data that enlarges and represents the first portion, and second enlarged image data that enlarges and represents a second portion that is included in an imaging range of the imaging device corresponding to the wide-angle image data and is different from the first portion, The moving image includes a first enlarged moving image created using the first enlarged image data and a second enlarged moving image created using the second enlarged image data. An image processing system according to any one of claims 24 to 26. 　 The image processing system according to any one of claims 24 to 27, wherein the calculation device stores the enlarged image data in a storage device, and creates the video using the stored enlarged image data based on the trigger information. 　 an imaging device that generates enlarged image data in a time series, the enlarged image data including wide-angle image data representing a situation including an object, first enlarged image data representing an enlarged first portion included in an imaging range corresponding to the wide-angle image data, and second enlarged image data representing an enlarged second portion included in an imaging range corresponding to the wide-angle image data and different from the first portion; a computing device that determines at least one of a first frame rate for generating the first enlarged image data in a time series manner and a second frame rate for generating the second enlarged image data in a time series manner based on the information of the first portion and the information of the second portion, and creates a first enlarged video using the first enlarged image data and a second enlarged video using the second enlarged image data; An image processing system comprising: 　 an imaging device that generates enlarged image data in a time series, the enlarged image data including wide-angle image data representing a situation including an object, first enlarged image data representing an enlarged first portion included in an imaging range corresponding to the wide-angle image data, and second enlarged image data representing an enlarged second portion included in an imaging range corresponding to the wide-angle image data and different from the first portion; creating a first enlarged video using the latest first enlarged image data at a plurality of first image selection timings, each of which is set at a first time interval, from among the first enlarged image data generated in time series; a computing device that creates a second enlarged video using the latest second enlarged image data at a plurality of second image selection timings that are set at second time intervals, out of the second enlarged image data that are generated in time series; An image processing system comprising: 　 The computing device creating the first enlarged video by using the latest first enlarged image data in a plurality of first search ranges that are respectively located a first search time before each of the plurality of first image selection timings, among the first enlarged image data that are generated in time series; creating the second enlarged video by using the latest second enlarged image data in a plurality of second search ranges preceding each of the plurality of second image selection timings by a second search time among the second enlarged image data generated in time series; 31. The image processing system of claim 30. 　 an imaging device for acquiring wide-angle image data representing a scene including an object; a calculation device that acquires enlarged image data that represents an enlarged first portion including at least a part of the object based on an imaging range of the imaging device that corresponds to the wide-angle image data, the computing device sets an object recognition area within the wide-angle image data that is used to acquire the enlarged image data, and acquires the enlarged image data when the first portion is included in the object recognition area; Image processing system. 　 The image processing system according to claim 32 , wherein the computing device does not acquire the enlarged image data if the first portion is not included in the object recognition region. 　 The image processing system described in any one of claims 1 to 33, wherein the imaging device comprises a light splitting element that splits incident light, a first optical system that forms an image of one of the light beams split by the light splitting element, a first imaging element that generates the wide-angle image data in accordance with the image formed by the first optical system, a second optical system that forms an image of the other light beam split by the light splitting element, a second imaging element that generates the enlarged image data in accordance with the image formed by the second optical system, and a modification element that changes the optical path of the other light beam so that the first portion is represented in the enlarged image data.

35. The image processing system of claim 34, wherein the modifying member comprises a mirror. 　 The image processing system described in any one of claims 1 to 35, wherein the imaging device has a light splitting member that splits incident light, a first optical system that forms an image of one of the light beams split by the light splitting member, a first imaging element that generates the wide-angle image data in accordance with the image formed by the first optical system, a second optical system that forms an image of the other light beam split by the light splitting member, a second imaging element that generates the enlarged image data in accordance with the image formed by the second optical system, and a modification member that changes at least one of the direction and position of the second imaging element so that the first portion is represented in the enlarged image data.

37. The image processing system according to claim 34, wherein the first optical system and the second optical system have at least a part in common. 　 The imaging device is a variable magnification optical system that forms an image of incident light and is capable of changing a focal length, and an image sensor that generates image data in accordance with an image formed by the variable magnification optical system, generating the wide-angle image data in accordance with an image formed when the focal length of the variable magnification optical system is a first focal length; generating the enlarged image data in accordance with an image formed in a state in which the focal length of the variable magnification optical system is a second focal length that is longer than the first focal length; An image processing system according to any one of claims 1 to 37. 　 The image processing system of any one of claims 1 to 38, wherein at least one of the imaging device and the arithmetic device creates the enlarged image data using a portion of the wide-angle image data that corresponds to the first portion. 　 An image processing device comprising a processor configured to generate still images representing the movement of at least one of a plurality of moving objects and a worker within a predetermined period of time, based on wide-angle image data representing the status of the objects and the status of a worker performing a process including a plurality of tasks while moving sequentially within a predetermined area for each of the plurality of objects. 　 a position of at least one of the plurality of objects and the worker is expressed as coordinates in a coordinate system having a predetermined position within the predetermined area as a reference point and two axes intersecting at the reference point; the processor generates, as the still image, a movement trajectory image including a two-dimensional graph in which one axis represents the elapsed time from a reference time for each of the processes and the other axis represents the coordinate values ​​of at least one of the two axes of the coordinate system corresponding to the positions of each of the plurality of objects and at least one of the workers when the elapsed time has elapsed from the reference time; The image processing device according to claim 40.

42. The image processing device according to claim 40, wherein the processor generates, as the still image, a stay duration image representing a distribution of stay durations, which are periods during which a change in position over time of at least one of the plurality of objects and the worker is smaller than a movement threshold, for each task included in the process when the process is performed for each of the plurality of objects. 　 The image processing device according to claim 42 , wherein the processor further generates, for each execution of the process on the plurality of objects, a duration-by-process image representing the duration of each of the tasks included in each process. 　 The image processing device described in claim 42 or 43, wherein the processor acquires enlarged image data based on the wide-angle image data, which enlarges and represents a first portion included in the imaging range of the wide-angle image data, and further generates, for each task included in the process, a detailed information image which chronologically represents information detected from the enlarged image data during the stay period at the task. 　 a position of at least one of the plurality of objects and the worker is expressed as coordinates in a coordinate system having a predetermined position within the predetermined area as a reference point and two axes intersecting at the reference point; the processor creates, as the still image, a heat map representing the cumulative stay time of the worker during the predetermined period for each coordinate in the coordinate system corresponding to the predetermined area. An image processing device according to any one of claims 40 to 44. 　 acquiring first wide-angle image data including an object at a first timing by an imaging device; acquiring, by a computing device, first enlarged image data that enlarges and represents a first portion including at least a part of the object at the first timing based on the first wide-angle image data; acquiring second wide-angle image data including the object at a second timing that is later than the first timing by the imaging device; when it is determined that the amount of change from the position of the first portion at the first timing to the position of the first portion at the second timing is within a change threshold using the second wide-angle image data, acquiring, by the arithmetic device, second enlarged image data that enlarges and represents the first portion at the second timing; An image processing method comprising:

47. The image processing method according to claim 46, further comprising creating, by the arithmetic device, an enlarged moving image that is a moving image of the first portion using the first enlarged image data and the second enlarged image data.

48. An image processing method as described in claim 46 or 47, wherein, when acquiring the second enlarged image data, if it is determined using the second wide-angle image data that the amount of change from the position of the first part at the first timing to the position of the first part at the second timing is outside a change threshold, the arithmetic device does not acquire second enlarged image data that enlarges and represents the first part at the second timing.

Images