Apparatus, method, and non-transitory computer readable medium

Abstract

An apparatus includes: an acquisition unit that acquires a plurality of images captured by changing a position of a focal plane with respect to a subject in an imaging direction; a selection unit that selects, from the plurality of images, two or more images having a contrast higher than a reference value; a synthesis unit that generates a synthesized image by synthesizing the two or more images; and a local image generation unit that generates, from each of the plurality of images, a local image of at least one position within the image and a position common to the plurality of images, and selects, from the plurality of images, two or more images having a contrast higher than the reference value. The selection unit selects, from a plurality of partial images at each position in the image, two or more partial images having a contrast higher than the reference value, and generates the synthesized image by synthesizing the two or more images. The compositing unit generates a composite partial image by compositing the two or more partial images at each position within the image.

Description

Technical Field

[0001] The present invention relates to apparatus, methods, and non-transitory computer-readable media. Background Art

[0002] Patent Document 1 and the like describe “combining an image object from a first optimal focal plane and an image object from a second optimal focal plane” (Claim 1 of Patent Document 1) and the like. Prior art literature Patent Document 1: Japanese Patent Application Laid-Open No. 2023-17808 Patent Document 2: U.S. Patent No. 8199997 Summary of the Invention

[0003] The device provided in the first embodiment of the present invention includes: an acquisition unit that acquires multiple images captured by changing the position of the focal plane relative to the subject in the shooting direction; a selection unit that selects two or more images with a contrast higher than a reference value from the multiple images; and a synthesis unit that generates a synthesized image by synthesizing the two or more images.

[0004] The above-mentioned device also has a local image generating unit, which generates local images of at least one position within the image and a common position among the multiple images from the multiple images, and selects two or more images with a contrast higher than the reference value from the multiple local images at each position within the image. As the composite image is generated by synthesizing the two or more images, the synthesizing unit generates the composite local image by synthesizing the two or more local images at each position within the image.

[0005] The above-mentioned apparatus further includes a size detection unit configured to detect a size of the subject in the image, and the partial image generation unit generates each partial image according to the size detected by the size detection unit.

[0006] Any of the above-mentioned devices including the partial image generating unit further includes a position detecting unit that detects the position of the subject in the image, and the partial image generating unit generates each partial image at the position detected by the position detecting unit.

[0007] In any of the above-mentioned devices having a local image generating unit, the local image generating unit generates a local image from the multiple images for each of the multiple positions within the image, and the selecting unit selects, for each of the multiple positions within the image, two local images that have a contrast higher than the reference value and are captured at two positions separated by a predetermined common interval in the shooting direction as focal planes.

[0008] In any of the above-mentioned devices having a local image generating unit, the local image generating unit generates a local image from the multiple images for each of the multiple positions within the image, the synthesis unit combines the local images for each position within the image and the local images including the synthesized local image, and performs brightness adjustment between the local images.

[0009] In the above-mentioned device, based on the fact that the contrasts of multiple local images generated by the local image generating unit for a position in the image are all below the reference value, the selecting unit selects a single local image from the multiple local images for the one position, and the synthesizing unit combines the local images and the local images including the single local image as a local image for the one position.

[0010] In any of the above-mentioned devices including the partial image generating unit, the acquiring unit acquires the plurality of images captured in a state where the plurality of subjects are arranged at positions different from each other in the capturing direction.

[0011] Any of the above-mentioned devices further includes an imaging unit that captures the plurality of images, and the acquisition unit acquires the plurality of images from the imaging unit.

[0012] The above-mentioned apparatus further includes an imaging control unit configured to change the position of the focal plane relative to the subject in the imaging direction and to cause the imaging unit to capture the plurality of images.

[0013] In the above-mentioned device, the imaging control unit reduces the interval between adjacent focal planes in the imaging direction and causes the imaging unit to re-capture the plurality of images in response to the imaging of an out-of-focus image.

[0014] In any of the above devices, the subject is a cell or a microorganism.

[0015] In the above-mentioned device, the subject is attached to a flexible and light-transmitting sheet-like member.

[0016] In any of the above-mentioned devices in which the subject is a cell or a microorganism, the subject is attached to the inner surface of the culture bag.

[0017] The method provided in the second embodiment of the present invention includes an acquisition stage, in which a plurality of images captured by changing the position of the focal plane relative to the subject in the shooting direction are acquired; a selection stage, in which two or more images having a contrast higher than a reference value are selected from the plurality of images; and a synthesis stage, in which a synthetic image is generated by synthesizing the two or more images.

[0018] A non-transitory computer-readable medium provided in a third aspect of the present invention stores a program, and a computer functions as an acquisition unit, a selection unit, and a synthesis unit by executing the program. The acquisition unit acquires multiple images captured by changing the position of the focal plane relative to the subject in the shooting direction; the selection unit selects two or more images with a contrast higher than a reference value from the multiple images; and the synthesis unit generates a synthesized image by synthesizing the two or more images.

[0019] In addition, the above summary of the invention does not list all the necessary features of the present invention. In addition, subcombinations of the above feature groups can also constitute inventions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A device 1 according to an embodiment is shown. Figure 2 Indicates the action of device 1. Figure 3 The culture bag 2 and the auxiliary parts 3 for recording are shown together. Figure 4 A cross section of the culture bag 2 and the auxiliary component 3 is shown. Figure 5 express Figure 4 A partial enlarged view of . Figure 6 Represents the local image before and after synthesis. Figure 7 The computer 1200 is shown as an example, on which various aspects of the present invention may be implemented in whole or in part. DETAILED DESCRIPTION

[0021] The present invention will be described below by way of embodiments of the invention, but the following embodiments are not intended to limit the scope of the invention. Furthermore, all combinations of features described in the embodiments are not necessarily essential solutions to the invention.

[0022] (Device 1) Figure 1 The device 1 according to this embodiment is shown. The device 1 includes an imaging unit 10 , an imaging control unit 11 , an acquisition unit 12 , a position detection unit 13 , a size detection unit 14 , a partial image generation unit 15 , a selection unit 16 , and a synthesis unit 17 .

[0023] ((Photography Unit 10)) The imaging unit 10 captures a plurality of images. Each image may include at least one subject, and in this embodiment, as an example, includes a plurality of subjects.

[0024] The subject is an object to be photographed, and in this embodiment, as an example, it can be a cell or a microorganism. The cells or microorganisms used as the subject can be used in the manufacture of regenerative medical products and cell medicines, and as an example, it can be iPS cells (201B7 strain). The subject can be attached to a flexible and light-transmitting thin sheet member, and as an example, it can be attached to a thin sheet of culture medium. The subject can be attached to the inner surface of a culture bag. The culture bag can culture cells or microorganisms attached to the inner surface, can be filled with culture fluid and maintain the interior as a closed environment. The culture bag can be made of soft resin.

[0025] The imaging unit 10 includes a microscope for magnifying and photographing a subject. The microscope may be any type of microscope, such as a bright-field microscope or a phase-contrast microscope. The imaging unit 10 can image a subject positioned on a support member (also called a stage) (not shown). Depending on the deflection and deformation of the sheet-like member or culture bag to which the subject is attached, the distance between the photographed subject and the stage may vary. The imaging unit 10 can supply each captured image to the acquisition unit 12.

[0026] ((Photography control unit 11)) The imaging control unit 11 controls the imaging unit 10 to capture multiple images (also referred to as focus bracket images) by changing the position of the focal plane relative to the subject in the imaging direction. The imaging control unit 11 can capture multiple focus bracket images by changing the position of the focal plane relative to the subject at reference intervals. The position of the focal plane relative to the subject can differ between the multiple focus bracket images.

[0027] The shooting direction may be the depth direction or the optical axis direction of the shooting unit 10. Changing the position of the focal plane relative to the subject in the shooting direction may mean changing the relative position of the subject and the focal plane in the shooting direction. The shooting control unit 11 may change the position of the focal plane relative to the subject by moving at least one of the optical system of the shooting unit 10 and the support member of the subject.

[0028] The imaging control unit 11 can capture multiple focus bracket images while maintaining the positions of the respective subjects, or can capture multiple focus bracket images with the subjects positioned at different positions in the imaging direction. The imaging control unit 11 controls the position of the focal plane so that one subject is captured in at least two focus bracket images.

[0029] ((Acquisition unit 12)) The acquisition unit 12 acquires a plurality of focus bracket images captured by changing the position of the focal plane relative to the subject in the imaging direction. The acquisition unit 12 may acquire the plurality of focus bracket images from the imaging unit 10. The acquisition unit 12 may supply the acquired plurality of focus bracket images to the position detection unit 13, the size detection unit 14, and the partial image generation unit 15, respectively.

[0030] ((Position Detection Unit 13)) The position detection unit 13 detects the position of the subject within the focus bracket image. As the subject's position, the position detection unit 13 detects the pixel position within the image. For example, it can detect the position represented by a coordinate system with the longitudinal and lateral axes of the image. In this embodiment, the subject's position can be the coordinate point of the subject's center or the coordinate range of the area occupied by the subject in the coordinate system. If multiple subjects exist within the focus bracket image, the position detection unit 13 can detect the position of each subject.

[0031] The position detection unit 13 can detect the position of each subject in each of the multiple focus bracket images. The position detection unit 13 can detect the position of each subject within the imaging area regardless of the imaging direction. If a subject is captured in two or more focus bracket images, the position detection unit 13 can detect a single position (for example, a position represented by average coordinates) as the position of the subject within these focus bracket images. The position detection unit 13 can supply the detected position of each subject to the partial image generation unit 15.

[0032] ((Size Detection Unit 14)) The size detection unit 14 detects the size of the subject within the focus bracket image. The size detection unit 14 may detect the subject's size as represented by the vertical and horizontal lengths of the image. If multiple subjects are present within the focus bracket image, the size detection unit 14 may detect the size of each subject.

[0033] The size detection unit 14 can detect the size of each subject in each of the multiple focus bracket images. If a single subject is captured in two or more focus bracket images, the size detection unit 14 can detect the largest size of the subject within these focus bracket images as the size of the single subject. The size detection unit 14 can supply the detected sizes of each subject to the partial image generation unit 15.

[0034] ((Partial Image Generator 15)) The partial image generating unit 15 generates a partial image for at least one position within the image, which is common to the plurality of focus bracket images, from each of the plurality of focus bracket images. As an example, the partial image generating unit 15 may generate a partial image for the center position of the image from each of the plurality of focus bracket images. The partial image generating unit 15 of this embodiment may generate a partial image from each of the plurality of focus bracket images for each of the plurality of positions within the focus bracket image.

[0035] The partial image generating unit 15 may generate each partial image according to the position detected by the position detecting unit 13. In this embodiment, as an example, the partial image generating unit 15 may generate a partial image according to a common position for each subject from a plurality of focus bracket images.

[0036] The partial image generation unit 15 can generate each partial image according to the size detected by the size detection unit 14. In this embodiment, as an example, the partial image generation unit 15 can generate a partial image of a common size for each subject from a plurality of focus bracket images. The partial image generation unit 15 can generate a partial image of a common size detected by the size detection unit 14 for each position detected by the position detection unit 13 from the plurality of focus bracket images. The size of the partial image can vary depending on the position within the image.

[0037] At least a portion of the subject can be captured in each partial image generated at a position detected by the position detection unit 13. Multiple partial images generated for the same position from different focus mount images can have different contrast, brightness, sharpness, and other characteristics. For example, when the focal plane is located near or at the attachment surface of the subject (in this embodiment, for example, the surface of the sheet-like member or the inner surface of the culture bag), the contrast of the partial image can be minimized. However, when the focal plane is moved to the inner or anterior side of this position, dark or bright contrast can be achieved. The subject can be in contact with the sheet-like member or the culture bag surface at the attachment surface, with the cross-section perpendicular to the imaging direction having the largest area. Dark contrast occurs when light passing through different portions of the specimen has opposite phases, weakening each other, resulting in a lower brightness than a first reference brightness. Bright contrast occurs when light passing through different portions of the specimen has the same phase, enhancing each other, resulting in a higher brightness than a second reference brightness. The first and second reference brightness can be set arbitrarily.

[0038] The partial image generation unit 15 may also generate partial images at positions where the subject is not detected by the position detection unit 13. The partial image generation unit 15 generates the partial images by removing the remaining portion of the focus bracket image from the partial images generated for the positions where the subject is detected. The contrast, brightness, sharpness, and other characteristics of the partial images at the positions where the subject is not detected can be substantially uniform.

[0039] The partial image generating unit 15 supplies the generated partial image for each position in the image to the synthesizing unit 17. As an example, the partial image generating unit 15 supplies each partial image to the selecting unit 16 in association with position information in the image.

[0040] ((Selection unit 16)) The selection unit 16 selects two or more focus bracket images having a contrast higher than a reference value from the plurality of focus bracket images. The selection unit 16 may select two focus bracket images from the plurality of focus bracket images acquired by the acquisition unit 12. In this embodiment, the selected images may be partial images, as an example. In this case, to select two or more images having a contrast higher than the reference value from the plurality of focus bracket images, the selection unit 16 may select two or more partial images having a contrast higher than the reference value from the plurality of partial images at respective positions within the image.

[0041] Contrast can be a value that represents the difference between light and dark in an image. For example, when the brightness of each pixel is represented by a pixel value, the maximum pixel value I in the image is used. max and the minimum pixel value I min And by contrast=(I max -I min ) / (I max +I min ) is used to calculate contrast. For example, the reference value may be 80% of the maximum possible pixel value. If the pixel value can take values ​​from 0 to 255, the reference value may be 204 (= 255 × 0.8). Furthermore, at the position within the image where the subject is located, the contrast in at least two partial images may be higher than the reference value. At the position within the image where the subject is not located, the contrast in each partial image may be lower than the reference value.

[0042] For locations within the plurality of partial images where the contrast is higher than a reference value, for example, locations where a subject is located within each location within the image, the selection unit 16 may select two partial images where the contrast is higher than the reference value. For example, for each of the plurality of locations within the image, the selection unit 16 may select two partial images captured with the focal planes of two locations separated by a predetermined common distance in the imaging direction and having the contrast higher than the reference value. The two partial images captured with the two locations as the focal planes may be generated from two images captured with the two locations as the focal planes.

[0043] The common interval can be an interval between two focal planes when capturing multiple focus bracket images, with at least one focal plane sandwiched between them, or it can be an interval obtained by multiplying the interval between adjacent focal planes in the shooting direction, i.e., a reference interval, by an integer greater than or equal to 2. As an example, the common interval can be four times the reference interval. In this case, the selection unit 16 can select, for a first position within the image, a focus bracket image captured with the n1th (n1 is a natural number) position from the back in the shooting direction as the focal plane, and a focus bracket image captured with the n1+4th position as the focal plane. For a second position within the image, the selection unit can select, for a second position within the image, a focus bracket image captured with the n2th (n2 is a natural number) position from the back in the shooting direction as the focal plane, and a focus bracket image captured with the n2+4th position as the focal plane.

[0044] The common interval can be set based on the size of the subject detected by the size detection unit 14, for example, half the average size of the subject, that is, an interval greater than the average radius of the cells or microorganisms that are the subject. In this case, the two selected partial images can be those captured with the front and back sides of the center of the subject as focal planes. When the focal plane moves, the two selected partial images can be partial images captured at a focal plane where the image has bright contrast or at focal planes before and after it, as well as partial images captured at a focal plane where the image has dark contrast or at focal planes before and after it. Based on the selection of one of the partial images, the selection unit 16 selects a partial image whose focal planes are separated by a common interval relative to the one partial image as the other partial image.

[0045] For locations where the contrast of multiple partial images is below a reference value, for example, locations where no subject is present within the image, the selection unit 16 may select a single partial image. In other words, if the contrast of multiple partial images generated by the partial image generation unit 15 for a single location within the image is below a reference value, the selection unit 16 may select a single partial image from the multiple partial images for that location. The selection unit 16 may select any one of the partial images.

[0046] The selection unit 16 supplies the selected partial images to the synthesis unit 17 according to their positions within the image. As an example, the selection unit 16 supplies the synthesis unit 17 with the partial images associated with the position information within the image.

[0047] ((Synthesis Department 17)) The synthesis unit 17 generates a synthesized image by synthesizing the two or more images selected by the selection unit 16. The synthesized image may be an image including a subject.

[0048] Combining two or more images to generate a composite image may involve generating a single composite image of the same size from two or more images of the same size. In this embodiment, as an example, the composited image may be a partial image. In this case, to generate a composite image by combining two or more images, the combining unit 17 combines two or more (two, as an example, in this embodiment) partial images selected by the selection unit 16 according to their positions within the image to generate a composite partial image.

[0049] The synthesis unit 17 synthesizes partial images so that each part of the subject (for example, intracellular organelles) is clearly captured. The synthesis unit 17 can generate a single partial image by deep synthesis of two or more partial images. This can be performed using a conventionally known image synthesis algorithm. Deep synthesis can generate a single image by extracting in-focus portions from multiple images with different focal plane positions and combining them.

[0050] The synthesis unit 17 detects the pixel with the highest sharpness from the two or more synthesized partial images according to the pixel position, and sets the pixel value of the detected pixel as the pixel value at that position in the synthesized image, thereby generating a synthesized image. As an example, the synthesis unit 17 can use the technology described in Patent Document 2 to synthesize the partial images. The sharpness of the pixel can be calculated using a conventionally known method. The synthesis unit 17 uses the image processing and image analysis software "CellPathfinder" (manufactured by Yokogawa Electric Corporation) to synthesize the partial images.

[0051] At least a portion of the subject is captured in the partial image generated by synthesis. The synthesis unit 17 may not perform synthesis of partial images at positions within the image where a single partial image has been selected by the selection unit 16, for example, at positions where the subject does not exist.

[0052] The synthesis unit 17 combines the partial images for respective positions within the image and adjusts the brightness between the partial images. The synthesis unit 17 adjusts the brightness of at least one partial image so that the brightness difference between the synthesized partial images falls within a reference range. The synthesis unit 17 may combine the partial images after adjusting their brightness, or may combine the partial images and then adjust the brightness of the partial images included in the combined image.

[0053] Here, combining partial images may refer to combining partial images with each other in the in-plane direction of the image to generate a larger image. The image generated by combining partial images may be an image of the same size as the original image (for example, the image captured by the capturing unit 10) that generates the partial image. Each combined partial image may include a synthesized partial image. The synthesizing unit 17 uses the partial image generated by the synthesis as the partial image of the combining object at the position where the two partial images selected by the selecting unit 16 are synthesized in each position in the image. The synthesizing unit 17 uses the single partial image as the partial image of the combining object at the position where a single partial image is selected by the selecting unit 16 in each position in the image. In other words, the synthesizing unit 17 can combine each combined partial image and each partial image including the single partial image as the partial image for the position where the single partial image is selected.

[0054] According to the above-described apparatus 1, a partial image is generated from each of a plurality of focus bracket images at at least one position within the image that is common to the plurality of focus bracket images. Two or more partial images having a contrast ratio higher than a reference value are selected from the plurality of partial images at each position in the image, and the partial images selected for each position in the image are synthesized. Therefore, even when photographing a subject whose position in the photographing direction is not fixed, a partial image with a high contrast ratio can be obtained for each position in the image.

[0055] Furthermore, since each partial image is generated with the size of the subject within the focus bracket image, a partial image with high contrast including the entire subject can be obtained.

[0056] Furthermore, since each partial image is generated at the position of the subject within the focus bracket image, a partial image that reliably includes the subject and has a high contrast can be obtained.

[0057] Furthermore, for each of multiple positions within the image, a partial image is generated from the multiple focus bracket images, and two partial images are selected, each captured at a focal plane with a contrast higher than a reference value and at two positions separated by a predetermined common distance in the shooting direction. Therefore, even when capturing subjects that are dispersed in the shooting direction and are of the same size, high-contrast images can be reliably obtained in each partial image. Furthermore, by selecting a partial image for each position within the image and determining the other partial image to be selected, the process of selecting two images for each position within the image can be simplified.

[0058] Furthermore, by combining partial images for respective positions within an image, including the partial images generated by synthesis, to generate a composite image, and performing brightness adjustment between the partial images, it is possible to prevent brightness dispersion among the partial images in the combined composite image.

[0059] Furthermore, based on the fact that the contrast of multiple partial images generated for a position within an image is all below a reference value, a single partial image is selected from the multiple partial images for that position, and the combined partial images are combined with the partial images including the single partial image as the partial image for that position. Therefore, for positions where no subject is present and the contrast of all the partial images in the focus bracket image is below the reference value, the process of combining the partial images can be omitted.

[0060] Furthermore, since a plurality of focus bracket images are acquired in a state where a plurality of subjects are arranged at mutually different positions in the imaging direction, it is possible to acquire partial images with high contrast for each subject dispersed in the imaging direction.

[0061] Furthermore, since a plurality of focus bracket images are acquired from the imaging unit 10 , a partial image with a high contrast can be acquired immediately, compared to a case where a plurality of focus bracket images are acquired from outside the apparatus.

[0062] Furthermore, since the imaging unit 10 is controlled to capture a plurality of focus bracket images by changing the position of the focal plane relative to the subject in the imaging direction, a high-contrast partial image can be reliably acquired.

[0063] Furthermore, since the subject is a cell or a microorganism, a high-contrast partial image of the cell or microorganism can be acquired.

[0064] Furthermore, since the subject is attached to a flexible and light-transmitting sheet-like member, a high-contrast partial image can be obtained for each cell or microorganism attached to the flexible sheet-like member and dispersed in the imaging direction.

[0065] Furthermore, since the subject is attached to the inner surface of the culture bag, a high-contrast local image can be obtained for each cell or microorganism dispersed in the imaging direction within the culture bag.

[0066] (Operation of Device 1) Figure 2 The following shows the operation of the device 1. The device 1 acquires a composite image by performing the processes of steps S11 to S19.

[0067] In step S11, the imaging unit 10 captures multiple focus bracket images. The imaging unit 10 captures multiple focus bracket images each time the imaging control unit 11 moves at least one of the optical system of the imaging unit 10 and the support member of the subject. The multiple captured focus bracket images are acquired by the acquisition unit 12.

[0068] Here, the imaging control unit 11 causes the imaging unit 10 to capture a plurality of focused bracket images while moving the focal plane in the imaging direction within a range in which at least a portion of the subject can be located. The imaging control unit 11 can cause the imaging unit 10 to capture a plurality of focused bracket images while moving the focal plane within a range that includes the attachment surface of the subject. For example, the imaging control unit 11 obtains from the user in advance the maximum height difference that may be caused by undulations on the inner surface of a sheet-like component or a culture bag to which cells or microorganisms as the subject are attached. The imaging control unit 11 causes the imaging unit 10 to capture a predetermined number (also referred to as a reference number) of focused bracket images while moving the focal plane within a range having an amplitude greater than the obtained maximum height difference.

[0069] In response to capturing an unfocused focus bracket image, the shooting control unit 11 reduces the interval between adjacent focal planes in the shooting direction and causes the shooting unit 10 to re-shoot a plurality of focus bracket images. An unfocused image may be an image that does not include a focused position. The shooting control unit 11 obtains a plurality of captured focus bracket images from the shooting unit 10 and determines whether each focus bracket image is in focus using a conventionally known focus determination method (for example, a frequency domain method, an edge detection method, a statistical method, etc.). The shooting control unit 11 may reduce the interval between the innermost focal plane and the most forward focal plane in the shooting direction and capture a reference number of focus bracket images, or may capture a greater number of focus bracket images than the reference number while maintaining the interval between the innermost focal plane and the most forward focal plane in the shooting direction.

[0070] In step S13, the position detection unit 13 and the size detection unit 14 detect the position and size of the object in the focus bracket image. The position detection unit 13 and the size detection unit 14 detect the object in the image using any method such as image recognition to detect its position and size.

[0071] In step S15, the partial image generating unit 15 generates a partial image for at least one position within each of the plurality of focus bracket images, which is a common position among the plurality of focus bracket images. In this embodiment, as an example, the partial image generating unit 15 generates a partial image of the detected size of the subject for each detected position of the subject, and generates the remaining portion after removing these partial images from the focus bracket image as a partial image for a position where the subject is not detected.

[0072] In step S17, the selection unit 16 selects two or more images with contrast higher than a reference value from the multiple focus bracket images. In this embodiment, as an example, the selection unit 16 selects two or more partial images with contrast higher than the reference value from the multiple partial images at various locations within the multiple focus bracket images. For locations with contrast higher than the reference value, such as the location of a subject, the selection unit 16 selects two partial images captured at two locations separated by a predetermined common distance in the shooting direction as focal planes. For locations where the contrast of the multiple partial images is lower than the reference value, such as locations where a subject is not present, the selection unit 16 selects a single partial image.

[0073] In step S19, the synthesis unit 17 synthesizes the two or more images to generate an image of the subject. In this embodiment, as an example, for each position within the image where two partial images were selected by the selection unit 16, such as the position where the subject is located, the synthesis unit 17 synthesizes the two partial images for each position, thereby generating a single partial image for that position. The synthesis unit 17 may not synthesize the partial images for each position within the image where a single partial image was selected by the selection unit 16, such as a position where the subject is not located.

[0074] The synthesis unit 17 combines the partial images for each position in the image to generate an image of the original image size, and adjusts the brightness between the partial images for each position, thereby obtaining an image in which multiple subjects are clearly captured.

[0075] According to the above operation, when an out-of-focus focus bracket image is captured, the intervals between adjacent focal planes are reduced in the shooting direction, and multiple focus bracket images are re-captured. Therefore, a focus bracket image can be obtained in which at least a portion of the focus bracket image is reliably focused. By using this focus bracket image for synthesis, a highly contrasted composite image can be reliably obtained.

[0076] (Cultivation bag 2 and auxiliary parts 3 for photographing) Figure 3 The culture bag 2 and the auxiliary parts 3 for recording are shown together.

[0077] The culture bag 2 has a flat hexagonal housing 20, and cultures cells or microorganisms attached to the inner surface of the housing 20. The interior of the housing 20 can be filled with a culture solution. The housing 20 can be formed of a gas-permeable film.

[0078] The auxiliary member 3 consists of two plate-like members 30 that can be opened and closed. Each plate-like member 30 has a light-transmitting, flat window 310 in the center. When the auxiliary member 3 is closed, holding the culture bag 2, the window 310 presses against the container 20 of the culture bag 2 from both sides, flattening the inner surface of the container 20.

[0079] Figure 4 The figure shows a cross section of the culture bag 2 and the auxiliary part. When the receiving portion 20 is held by the auxiliary part 3, a plate-shaped perforated spacer 32 is provided between the window portion 310 of the auxiliary part 3 and the receiving portion 20 of the culture bag 2. The perforated spacer 32 has a plurality of through-hole portions 320 extending in the thickness direction, which are used to ensure the permeability of gas inside and outside the receiving portion 20. The plurality of through-hole portions 320 can be formed into rectangular shapes, or can be arranged in a grid shape as a whole. As an example, when the photographing unit 10 photographs a 20-fold magnified image, the through-hole portion 320 can be formed into a square with a side length of 2.5 mm. The local image generated by the local image generating unit 15 is smaller than the through-hole portion 320.

[0080] Figure 5 Expressed Figure 4 A partially enlarged view. The housing 20, held by the auxiliary member 3, can bulge outward at the location of the through-hole 320 of the aperture spacer 32. This causes the inner surface of the housing 20 to deflect unevenly by approximately ±30 μm in the imaging direction (the vertical direction in the figure). In this case, as an example, the imaging control unit 11 moves the focal plane 1 μm at a time, capturing a total of 61 focus holder images.

[0081] (Image example) Figure 6 The figure shows partial images before and after synthesis. In the figure, the two upper partial images are partial images used for synthesis and represent two partial images selected by the selection unit 16 from the partial images generated for each position from multiple focus bracket images captured by the imaging unit 10. In the figure, the upper left partial image is a bright-contrast partial image, and the upper right partial image is a dark-contrast partial image. The lower image in the figure shows a synthesized partial image synthesized from these two partial images. As shown in the figure, the apparatus 1 can obtain clear images of various parts of the subject.

[0082] (Variation) In the above embodiment, the size detection unit 14 supplies the size of each subject to the partial image generation unit 15. However, a single common size may be supplied to the partial image generation unit 15. The common size may be the largest of the detected sizes. In this case, the partial image generation unit 15 generates a partial image of the common size according to the positions of the subject detected by the position detection unit 13.

[0083] Furthermore, although the partial image generating unit 15 has been described as generating a partial image even at a position where no subject is detected, it is not necessary to generate a partial image. In this case, the synthesizing unit 17 does not need to combine the synthesized partial images.

[0084] Furthermore, while the selection unit 16 has been described as selecting two partial images captured at two locations within the image, each with a contrast higher than a reference value and separated by a common interval in the shooting direction, the selection unit 16 may also select two partial images captured at two locations separated by a different interval in the shooting direction. For example, for each position of an object detected by the position detection unit 13, the selection unit 16 may select two partial images captured at two locations separated by a distance in the shooting direction corresponding to the object size detected by the size detection unit 14. The distance corresponding to the object size may be half the object size, that is, a distance greater than the radius of the cell or microorganism being the object.

[0085] Furthermore, while the apparatus 1 has been described as including the imaging unit 10 and the imaging control unit 11, it is not necessary to include either of these components. If the apparatus 1 does not include the imaging unit 10, the acquisition unit 12 can acquire multiple focus bracket images captured by an imaging device external to the apparatus 1. If the apparatus 1 does not include the imaging control unit 11, an operator can manually control the imaging unit 10 to capture multiple focus bracket images.

[0086] Furthermore, while the apparatus 1 has been described as including the position detection unit 13, the size detection unit 14, and the partial image generation unit 15, it is not required to include any of these components. If the apparatus 1 does not include the position detection unit 13, the partial image generation unit 15 can generate partial images from each focus bracket image in a predetermined manner. For example, the partial image generation unit 15 can obtain partial images of each area enclosed by vertical and horizontal lines arranged in a grid pattern within the image. The size of the partial image can be the average or maximum value of the sizes of each subject detected by the size detection unit 14, or it can be a predetermined size. If the apparatus 1 does not include the size detection unit 14, the partial image generation unit 15 can generate a partial image of a predetermined size. If the partial image generation unit 15 generates a partial image of a predetermined size, the apparatus 1 can include a setting unit for resizing the partial image. If the synthesized partial image synthesized by the synthesis unit 17 is not in focus on the subject, the setting unit reduces the size of the partial image and re-executes steps S15 to S19.

[0087] Furthermore, while the apparatus 1 has been described as including the partial image generating unit 15, it is not necessary to include the partial image generating unit 15. If the apparatus 1 does not include the partial image generating unit 15, the selecting unit 16 can select two or more images having a contrast higher than a reference value from the plurality of focus bracket images, and the synthesizing unit 17 can synthesize the two or more selected images to generate a composite image. This allows for obtaining a high-contrast composite image even when capturing a subject whose position in the capturing direction is not constant.

[0088] Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, where a module may represent (1) a stage of a process for performing an operation or (2) a portion of a device having the function of performing an operation. Specific stages and portions may be implemented by dedicated circuits, programmable circuits supplied with computer-readable instructions stored on a computer-readable medium, and / or processors supplied with computer-readable instructions stored on a computer-readable medium. Dedicated circuits may include digital and / or analog hardware circuits, and may also include integrated circuits (ICs) and / or discrete circuits. Programmable circuits may include reconfigurable hardware circuits that include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and other memory elements.

[0089] A computer-readable medium may include any tangible device capable of storing instructions for execution by an appropriate device. Consequently, a computer-readable medium having instructions stored therein includes an article containing instructions that can be executed to produce a means for performing the operations specified by the flowchart or block diagram. Examples of computer-readable media include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media include floppy disks, magnetic disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile discs (DVD), Blu-ray discs, memory sticks, integrated circuit cards, and the like.

[0090] Computer-readable instructions include any of source code and object code described by any combination of one or more programming languages ​​including assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or object-oriented programming languages ​​such as Smalltalk (registered trademark), JAVA (registered trademark), C++, and existing procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0091] Computer-readable instructions can be provided to a processor or programmable circuit of a programmable data processing device such as a computer via a local or local area network (LAN), a wide area network (WAN) such as the Internet, and the computer-readable instructions are executed in order to make a unit for performing the operations specified by the flowchart or block diagram. Here, the computer can be a PC (personal computer), a tablet computer, a smart phone, a workstation, a server computer, a general-purpose computer or a special-purpose computer, etc., or a computer system connected to multiple computers. Such a computer system connected to multiple computers is also called a distributed computing system, which is a computer in a broad sense. In a distributed computing system, multiple computers respectively execute parts of a program, and data in the execution of the program is handed over between the computers as needed, so that multiple computers execute the program centrally.

[0092] Examples of processors include computer processors, central processing units (CPUs), processing units, microprocessors, digital signal processors, controllers, and microcontrollers. A computer may have one processor or multiple processors. In a multi-processor system with multiple processors, each processor executes a portion of a program, and data in program execution is transferred between processors as needed, thereby allowing multiple processors to execute the program centrally. For example, in the execution of multiple tasks, multiple processors can each execute a portion of each task in a subdivided manner by switching tasks in each time slice. In this case, which portion of a program is executed by each processor changes dynamically. Which portion of a program is executed by each processor can also be statically determined by being aware of the programming of the multiple processors.

[0093] Figure 7 This figure illustrates an example of a computer 1200 capable of implementing various aspects of the present invention in whole or in part. Programs installed on computer 1200 enable computer 1200 to perform operations associated with devices according to embodiments of the present invention or functions of one or more components of such devices, or to execute such operations or such one or more components, and / or enable computer 1200 to perform processes according to embodiments of the present invention or stages of such processes. Such programs can be executed by CPU 1212 to cause computer 1200 to perform specific operations associated with some or all of the modules in the flowcharts and block diagrams described in this specification.

[0094] The computer 1200 of this embodiment includes a CPU 1212, a RAM 1214, a graphics controller 1216, and a display device 1218, which are interconnected via a main controller 1210. The computer 1200 also includes a communication interface 1222, a storage device 1224 such as a hard disk drive, an input / output unit such as a DVD-ROM drive 1226, and an IC card drive, which are connected to the main controller 1210 via an input / output controller 1220. The computer also includes a conventional input / output unit such as a ROM 1230 and a keyboard 1242, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0095] The CPU 1212 controls each unit by operating according to programs stored in the ROM 1230 and the RAM 1214. The graphics controller 1216 acquires image data generated by the CPU 1212 from a frame buffer or the like provided in the RAM 1214 or from the graphics controller 1216 itself, and displays the image data on the display device 1218.

[0096] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 within the computer 1200. The DVD-ROM drive 1226 reads programs and data from the DVD-ROM 1227 and provides the programs and data to the storage device 1224 via the RAM 1214. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0097] The ROM 1230 stores therein a boot program or the like executed by the computer 1200 upon activation and / or programs that depend on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0098] The program is provided on a computer-readable medium such as a DVD-ROM 1227 or an IC card. The program is read from the computer-readable medium, installed in the storage device 1224, RAM 1214, or ROM 1230, also examples of computer-readable media, and executed by the CPU 1212. The information processing described in these programs is read into the computer 1200, thereby enabling the program to cooperate with the various types of hardware resources described above. A device or method can be constructed by implementing information manipulation or processing in conjunction with the use of the computer 1200.

[0099] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 can execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, the DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area provided on the recording medium.

[0100] Furthermore, the CPU 1212 can read all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, the DVD-ROM drive 1226 (DVD-ROM 1227), or an IC card into the RAM 1214, and perform various types of processing on the data in the RAM 1214. The CPU 1212 then writes the processed data back to the external recording medium.

[0101] Various types of information such as various types of programs, data, tables, and databases can be stored in a recording medium and subjected to information processing. The CPU 1212 performs various types of processing described in various places of this disclosure on the data read from the RAM 1214 and writes the results back to the RAM 1214. The various types of processing include various types of operations specified by the instruction sequence of the program, information processing, conditional judgment, conditional branching, unconditional branching, information retrieval / replacement, etc. In addition, the CPU 1212 can retrieve information from files, databases, etc. in the recording medium. For example, in the case where a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 can retrieve an entry that is consistent with the condition specifying the attribute value of the first attribute from the plurality of entries, and read the attribute value of the second attribute stored in the entry, thereby obtaining the attribute value of the second attribute associated with the first attribute that meets the predetermined condition.

[0102] The programs or software modules described above may be stored in a computer-readable medium on or near the computer 1200. Furthermore, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as the computer-readable medium, thereby providing the program to the computer 1200 via the network.

[0103] While the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various modifications or improvements can be made to the above embodiments. As can be seen from the claims, embodiments with such modifications or improvements are also included in the technical scope of the present invention.

[0104] The order of execution of actions, processes, steps, and stages, etc., in the apparatus, system, program, and method described in the claims, specifications, and drawings is not specifically indicated as "before," "before," or the like. Furthermore, it should be noted that the actions, processes, steps, and stages may be executed in any order as long as the output of the previous process is not used in the subsequent process. Even if the action flow in the claims, specifications, and drawings is described using the phrases "first," "next," or the like for ease of explanation, it does not necessarily mean that the actions must be executed in that order. Description of Reference Numerals

[0105] 1 device, 2 culture bag, 3 auxiliary part, 10 imaging unit, 11 imaging control unit, 12 acquisition unit, 13 position detection unit, 14 size detection unit, 15 local image generation unit, 16 selection unit, 17 synthesis unit, 20 storage unit, 30 plate-shaped member, 32 open hole spacer, 310 window portion, 320 through-hole portion, 1200 computer, 1210 main controller, 1212 CPU, 1214 RAM, 1216 graphics controller, 1218 display device, 1220 input / output controller, 1222 communication interface, 1224 storage device, 1226 DVD-ROM drive, 1227 DVD-ROM, 1230 ROM, 1240 input / output chip, 1242 keyboard.

Claims

1. A device, characterized in that include: an acquisition unit that acquires a plurality of images captured by changing the position of the focal plane relative to the subject in a capturing direction; a selection unit for selecting two or more images having a contrast ratio higher than a reference value from the plurality of images; as well as The synthesis unit generates a synthesized image by synthesizing the two or more images.

2. The device according to claim 1, characterized in that The apparatus further comprises a partial image generating unit configured to generate, from each of the plurality of images, a partial image of at least one position within the image and a position common to the plurality of images. The selecting unit selects two or more images having a contrast higher than the reference value from the plurality of images, wherein the selecting unit selects two or more partial images having a contrast higher than the reference value from the plurality of partial images at respective positions within the image. As generating the composite image by combining the two or more images, the combining unit generates the composite partial image by combining the two or more partial images at respective positions within the image.

3. The device according to claim 2, characterized in that further comprising a size detection unit configured to detect the size of the subject within the image, The partial image generating unit generates each partial image according to the size detected by the size detecting unit.

4. The device according to claim 2, characterized in that further comprising a position detection unit configured to detect the position of the subject within the image, The partial image generating unit generates each partial image at the position detected by the position detecting unit.

5. The device according to claim 2, characterized in that The partial image generating unit generates a partial image from the plurality of images for each of the plurality of positions within the image. The selection unit selects, for each of a plurality of positions within an image, two partial images having a contrast higher than the reference value and captured with two positions separated by a predetermined common interval in the imaging direction as focal planes.

6. The device according to claim 2, characterized in that The partial image generating unit generates a partial image from the plurality of images for each of the plurality of positions within the image. The synthesis unit combines partial images corresponding to respective positions in an image and including the synthesized partial image, and performs brightness adjustment between the partial images.

7. The device according to claim 6, characterized in that The selection unit selects a single partial image from the plurality of partial images for a position in the image, based on the fact that the contrasts of all the plurality of partial images generated by the partial image generation unit for the position are below the reference value. The synthesis unit combines the partial images including the single partial image as the partial image for the one position.

8. The device according to claim 2, characterized in that The acquisition unit acquires the plurality of images captured in a state where the plurality of subjects are arranged at positions different from each other in the imaging direction.

9. The device according to claim 1, characterized in that further comprising a photographing unit for photographing the plurality of images, The acquisition unit acquires the plurality of images from the imaging unit.

10. The device according to claim 9, characterized in that The imaging control unit is further provided. The imaging control unit changes the position of the focal plane relative to the subject in the imaging direction and causes the imaging unit to capture the plurality of images.

11. The device according to claim 10, characterized in that The imaging control unit reduces the interval between adjacent focal planes in the imaging direction and causes the imaging unit to re-capture the plurality of images in response to the imaging of an out-of-focus image.

12. The device according to claim 1, characterized in that The subject is a cell or a microorganism.

13. The device according to claim 12, characterized in that The object is attached to a flexible and light-transmitting sheet-like member.

14. The device according to claim 12, characterized in that The object is attached to the inner surface of the culture bag.

15. A method, characterized in that include: an acquisition stage, acquiring a plurality of images captured by changing the position of the focal plane relative to the subject in a shooting direction; In a selection stage, two or more images having a contrast ratio higher than a reference value are selected from the plurality of images; and In the synthesis stage, a synthesized image is generated by synthesizing the two or more images.

16. A non-transitory computer-readable medium, characterized in that A program is stored, and the computer functions as an acquisition unit, a selection unit, and a synthesis unit by executing the program. The acquisition unit acquires a plurality of images captured by changing the position of the focal plane relative to the subject in the capturing direction. The selection unit selects two or more images with a contrast ratio higher than a reference value from the plurality of images. The synthesis unit generates a synthesized image by synthesizing the two or more images.

Images