Image processing device

Abstract

The present invention provides an image processing device that makes it possible to reduce the influence of noise added to a region that is subject to recognition in a captured image. An image processing device according to the present invention is provided with: an image sensor (20) that captures an image of a subject of recognition; an image recognition unit (120) that identifies a recognition subject region (610) of the image; an image noise determination unit (122) that determines the presence or absence of image noise in the recognition subject region (610) and that calculates a noise-added region (611); and an avoidance method calculation unit (123). When the image noise determination unit (122) has detected image noise, the position of the image sensor (20) is changed. The image sensor (20) at the changed position captures an image. The image recognition unit (120) identifies a recognition subject region (610) of the image. The image noise determination unit (122) calculates a noise-added region (611) of the image. The avoidance method calculation unit (123) calculates a method for changing the position of the image sensor (20) for avoiding image noise on the basis of the changes in the positions of the image sensor (20) and the noise-added region (611).

Description

Image Processing Device

[0001] The present invention relates to an image processing device for processing captured images, in particular images captured through a transparent object.

[0002] Currently, automation of work in laboratories for in vitro diagnostics is progressing. As one example of such work automation, the use of robots for tasks such as replacing consumables for in vitro diagnostic equipment is being considered. Automating the replacement of consumables requires a robot to appropriately select, hold, and transport the necessary consumables from a storage location to replenish them in the target equipment.

[0003] A refrigerator is an example of a place where such consumables are stored. Refrigerators installed in examination rooms generally have transparent doors such as glass doors, and contain consumables and other items required for examinations. It is desirable to recognize and check the items stored in the refrigerator with the refrigerator door closed, without opening it, in order to prevent temperature changes in the stored items.

[0004] Robots in laboratories are required to perform their work without interfering with the work of the laboratory technicians as much as possible. For this reason, the robot needs to be able to recognize and check the items stored in the refrigerator as quickly as possible and reduce the amount of time it spends in front of the refrigerator. Therefore, to automate the replacement of consumables with robots, especially the recognition and confirmation of items stored in the refrigerator, it is effective to apply image recognition processing technology using an image sensor that can recognize items with the refrigerator's glass door closed and can individually recognize multiple items at once.

[0005] In such image recognition processing technology, it is important to prevent recognition failures due to noise in the image caused by the glass door.Noise caused by the glass door includes noise caused by foreign objects reflected on the glass door (e.g., light, equipment and people in the examination room), and foreign objects attached to the glass door itself (e.g., dirt, fogging, etc.).

[0006] In order to prevent image recognition failures due to noise, various techniques have been developed, such as the technique described in Patent Document 1. Patent Document 1 describes an image restoration method in which a first image of an object that is optically occluded by contamination is captured, a second image of the object is captured from a different viewpoint, and the optically occluded portion of the first image is reconstructed using information from the second image.

[0007] International Publication No. 2010 / 084707

[0008] In conventional technologies, for example, in a recognition process of an image of an item stored in a refrigerator, sufficient consideration has not been given to preventing failure in recognition of the stored item due to noise in the captured image caused by the glass door of the refrigerator. For example, in the technology described in Patent Document 1, in both the first image and the second image, consideration is not given to preventing the area to be recognized in the image from being affected by noise caused by the glass door.

[0009] An object of the present invention is to provide an image processing device that can reduce the influence of noise added to an area to be recognized within a captured image.

[0010] The image processing device according to the present invention comprises an image sensor that captures an image of an object to be recognized, a drive mechanism in which the image sensor is installed and which can change the position and attitude of the image sensor, a control unit that controls the drive mechanism to change the position and attitude of the image sensor, an image recognition unit that performs image recognition processing on the image captured by the image sensor and identifies a recognition target area which is an area in the image from which information is acquired, an image noise determination unit that determines the presence or absence of image noise in the recognition target area based on changes in luminance distribution, and if the image noise is detected, calculates the position and size of a noise-added area which is an area in the recognition target area to which the image noise is added, and an avoidance method calculation unit that calculates a method for changing the position and attitude of the image sensor to reduce the effects of the image noise. When the image noise determination unit detects the image noise in the recognition target area in the first image captured by the image sensor, the control unit changes the position and orientation of the image sensor, the image sensor whose position and orientation have been changed captures the second image, the image recognition unit identifies the recognition target area in the second image, the image noise determination unit calculates the position and size of the noise-added area in the recognition target area in the second image, the avoidance method calculation unit calculates the change method based on the change in position and orientation of the image sensor and the change in position of the noise-added area in the second image from the first image, and the control unit changes the position and orientation of the image sensor in accordance with the change method calculated by the avoidance method calculation unit.

[0011] According to the present invention, it is possible to provide an image processing device that can reduce the influence of noise added to an area to be recognized within a captured image.

[0012] 7 is a configuration diagram of an image processing device according to a first embodiment of the present invention. FIG. 1 is a diagram illustrating an example of stored items that are the object of image capture by an image sensor and a storage location where the stored items are stored in the first embodiment. FIG. 2 is a flowchart illustrating a procedure by which the image processing device according to the first embodiment acquires individual information of stored items in a refrigerator. FIG. 3 is a top view illustrating a positional relationship between the image sensor and the stored items that are the object of recognition in the first embodiment. FIG. 4 is a flowchart illustrating a procedure of image recognition processing executed by the image processing device according to the first embodiment. FIG. 5 is a diagram illustrating an example of an image captured by the image sensor of the image processing device according to the first embodiment. FIG. 6 is a diagram illustrating an example of a method by which an image noise determination unit calculates the lateral position and size of a noise-added region in the first embodiment. FIG. 7 is a top view illustrating a positional relationship between the image sensor and the stored items that are the object of recognition when an avoidance method calculation unit calculates a method of changing the position and attitude of the image sensor in the first embodiment. FIG. 8 is a diagram illustrating an example of an image captured by the image sensor of the image processing device according to the first embodiment, and is an example of an image captured when the position of the image sensor is changed as shown in FIG. 7. FIG. 9 is a diagram illustrating an example of an image captured by the image sensor of the image processing device according to the first embodiment, and is an example of an image captured when the position of the image sensor is changed in S507 of FIG. Fig. 10 is a diagram showing an example of a display screen output by a display unit in the image processing device according to Example 1. Fig. 11 is a flowchart showing the procedure of image recognition processing executed by the image processing device according to Example 2 of the present invention. Fig. 12 is a flowchart showing the procedure of image recognition processing executed by the image processing device according to Example 3 of the present invention. Fig. 13 is a flowchart showing the procedure of acquiring individual information of items stored in a refrigerator executed by the image processing device according to Example 4 of the present invention.

[0013] The image processing device according to the present invention can avoid noise in the captured image and reduce the effect of noise by moving noise added to an area to be recognized in the captured image to an area other than the area to be recognized. When processing an image captured through a transparent object, the image processing device according to the present invention can avoid noise in the image caused by the transparent object while maintaining the angle of view and quality of the image required for image recognition, thereby preventing failure in recognizing the object to be recognized and improving the recognition rate.

[0014] An image processing device according to an embodiment of the present invention will be described below with reference to the drawings. In the drawings used in this specification, identical or corresponding components are designated by the same reference numerals, and repeated description of these components may be omitted. Furthermore, when there are multiple identical components and they are to be described separately, an alphabet (e.g., a, b, c) is added to the end of the reference numeral. When it is not necessary to distinguish between the individual components, the components will be described without the alphabet.

[0015] In the following description, reducing the effect of noise in a captured image is referred to as “avoiding noise.” Furthermore, noise in a captured image is referred to as “image noise” or simply “noise.”

[0016] An image processing apparatus according to a first embodiment of the present invention will be described.

[0017] 1 is a block diagram of an image processing apparatus 1 according to this embodiment. The image processing apparatus 1 includes a mobile carriage 10, a drive mechanism 11, an image sensor 20, and a control device 100.

[0018] The movable carriage 10 includes a drive mechanism 11 and wheels. The movable carriage 10 can change its position by driving the wheels to move forward, backward, left, and right.

[0019] The drive mechanism 11 includes multiple drive shafts and an end drive mechanism 12. The end drive mechanism 12 is a mechanism for manipulating and gripping a work object. The drive mechanism 11 can change the position and posture of the end drive mechanism 12 by operating the multiple drive shafts.

[0020] The image sensor 20 is installed on the end drive mechanism 12 and captures an image of a recognition target. The image sensor 20 can change its position and orientation by the operation of the drive mechanism 11.

[0021] The control device 100 can be configured as a computer, is connected to the mobile cart 10 and the drive mechanism 11, and controls the driving of the mobile cart 10 and the drive mechanism 11. The control device 100 is also connected to the image sensor 20, acquires an image of the object to be recognized from the image sensor 20, and performs recognition processing on this image.

[0022] The control device 100 includes a sequence control unit 110, an operation control unit 111, an image recognition unit 120, an image acquisition unit 121, an image noise determination unit 122, an avoidance method calculation unit 123, an input / output processing unit 130, an input unit 131, and a display unit 132.

[0023] The sequence control unit 110 controls the overall operation of the image processing device 1. For example, the sequence control unit 110 can control the drive mechanism 11 to change the position and attitude of the image sensor 20. The sequence control unit 110 also includes a storage device and can store data necessary for controlling the operation of the image processing device 1.

[0024] The operation control unit 111 is connected to the mobile carriage 10 , the drive mechanism 11 , and the sequence control unit 110 , and drives the mobile carriage 10 and the drive mechanism 11 based on instructions from the sequence control unit 110 .

[0025] The image recognition unit 120 is connected to the sequence control unit 110 and performs various calculation processes for image recognition processing based on instructions from the sequence control unit 110.

[0026] The image acquisition unit 121 is connected to the image sensor 20 and the image recognition unit 120 , acquires an image captured by the image sensor 20 from the image sensor 20 , and outputs the image to the image recognition unit 120 .

[0027] The image noise determination unit 122 is connected to the sequence control unit 110 and the image recognition unit 120, and determines whether or not noise exists in the image area recognized by the image recognition unit 120.

[0028] The avoidance method calculation unit 123 is connected to the sequence control unit 110 and the image noise determination unit 122, and when the image noise determination unit 122 determines that noise exists in the image area recognized by the image recognition unit 120, it calculates a method for reducing the effect of noise in the image recognition process (a method for avoiding noise).

[0029] The input / output processing unit 130 is connected to the sequence control unit 110, the image recognition unit 120, the image noise determination unit 122, and the avoidance method calculation unit 123, and inputs the setting values ​​required for each process and outputs the results of each process.

[0030] The input unit 131 is configured with input devices such as a keyboard and a mouse, and is connected to the input / output processing unit 130. The input unit 131 inputs information required for processing by the image processing device 1, for example.

[0031] The display unit 132 is configured with output devices such as a monitor and a speaker, and is connected to the input / output processing unit 130. The display unit 132 outputs a display screen. This display screen includes, for example, a setting input screen for inputting information necessary for processing by the image processing device 1, and a result display screen for outputting the processing results of the image processing device 1.

[0032] 2 is a diagram showing an example of a stored item 210 to be imaged by the image sensor 20 and a storage location where the stored item 210 is stored in this embodiment. In this embodiment, as an example, the storage location is a refrigerator 200 installed in an examination room.

[0033] The stored items 210 (210a to 210e) are objects stored in the refrigerator 200, which is a storage location, and are, for example, containers containing reagents. The control device 100 performs recognition processing on images of the stored items 210. In addition to the refrigerator 200, the storage location can also include, for example, a warming cabinet and a storage shelf.

[0034] Individual information about the stored item 210 is written on the side or top surface of the stored item 210. For example, the stored item 210 may have a surface to which a label is attached, and the individual information is written on this label, or the stored item 210 may have a surface on which the individual information is written directly. The individual information about the stored item 210 is information for identifying the stored item 210 or information about the attributes of the stored item 210, and may include, for example, the identification number, type, and expiration date of the stored item 210.

[0035] Refrigerator 200 has a door, at least a portion of which is made of a transparent material (for example, glass or a resin such as plastic). In this embodiment, as an example, refrigerator 200 has glass door 201, at least a portion of which is transparent. Refrigerator 200 stores items 210a to 210e inside. In the example shown in FIG. 2 , refrigerator 200 has two shelves inside, with items 210a to 210c stored on the upper shelf and items 210d and 210e stored on the lower shelf.

[0036] The refrigerator 200 does not necessarily have to have a door.

[0037] The mobile cart 10 (FIG. 1) of the image processing device 1 can move within the examination room in which the refrigerator 200 is installed.

[0038] FIG. 3 is a flowchart showing a procedure for the image processing device 1 according to this embodiment to acquire individual information of the stored item 210 in the refrigerator 200.

[0039] It is assumed that the mobile cart 10 of the image processing device 1 moves within the examination room and stops in front of the refrigerator 200.

[0040] In S301, the image processing device 1 operates the drive mechanism 11 through processing by the sequence control unit 110 to move the image sensor 20. The image sensor 20 moves to a position and posture that allows it to capture an image of the stored item 210. The image capture target of the image sensor 20 is the stored item 210.

[0041] The positional relationship between the image sensor 20, the refrigerator 200, and the stored item 210, which is the object to be recognized, will be described with reference to FIG.

[0042] FIG. 4 is a top view showing the positional relationship between the image sensor 20 and stored items 210a to 210c, which are objects to be recognized, in this embodiment.

[0043] Stored items 210a to 210c are stored in refrigerator 200, lined up in the following order. It is desirable that stored items 210a to 210c are stored in a position and orientation such that the surfaces that require image recognition are generally directly facing glass door 201 of refrigerator 200. In this embodiment, stored items 210a to 210c are stored in refrigerator 200 in such a position and orientation. The surfaces of stored items 210a to 210c that require image recognition are surfaces on which individual information of stored items 210a to 210c is printed.

[0044] Image sensor 20 moves to a position and posture that generally faces stored items 210a to 210c stored inside refrigerator 200. Image sensor 20 is positioned and oriented so that the center of its optical axis is generally aligned with the vicinity of the center of stored item 210b and so that stored items 210a to 210c are within its field of view through glass door 201. Such a position and posture of image sensor 20 is preset in sequence control unit 110 as teaching data.

[0045] As shown in Fig. 4, a coordinate system 400 is set in the inspection room in which refrigerator 200 is installed. The x-axis direction of coordinate system 400 is the width direction of refrigerator 200, the y-axis direction is the depth direction of refrigerator 200, and the z-axis direction is the height direction of refrigerator 200.

[0046] Returning to the description of FIG.

[0047] In S302, the image processing device 1 executes image recognition processing to acquire individual information of the stored items 210a to 210c.

[0048] The process of S302 will be described with reference to FIG.

[0049] FIG. 5 is a flowchart showing the procedure of the image recognition process executed by the image processing device 1 according to this embodiment.

[0050] In S501, image sensor 20 sandwiches glass door 201, which is a transparent body, between itself and stored item 210, and captures an image of stored item 210 (a first image of stored item 210) through glass door 201. If refrigerator 200 does not have a door, image sensor 20 captures an image of stored item 210 without passing through a transparent body. Image acquisition unit 121 of control device 100 acquires the image captured by image sensor 20 from image sensor 20.

[0051] In S502, the image recognition unit 120 acquires an image from the image acquisition unit 121, performs image recognition processing, and identifies a recognition target area 610 (recognition target areas 610a to 610c shown in FIG. 6) in the acquired image. The recognition target area 610 is an area in the image of the stored item 210 that is the target of image recognition and from which information is acquired. The recognition target area 610 will be described later.

[0052] In S503 , the image noise determination unit 122 determines whether or not image noise exists in the recognition target area 610 .

[0053] Here, the details of the processes in S502 and S503 will be explained with reference to FIGS. 6A and 6B.

[0054] 6A is a diagram showing an example of an image (captured image 600a) captured by the image sensor 20 of the image processing device 1 according to this embodiment. The captured image 600a includes images of the stored items 210a to 210c described with reference to FIG.

[0055] In the captured image 600a, the areas containing the individual information of the stored items 210a to 210c are called recognition target areas 610a to 610c. Figure 6A shows the recognition target areas 610a to 610c of the stored items 210a to 210c, respectively. The recognition target areas 610a to 610c are specified by their position and size (width and height) in the captured image 600a.

[0056] As shown in FIG. 6A, the position of the recognition target region 610b is P(w ｐ , h ｐ ) and the size is expressed as width W ｐ and height H ｐIt is also assumed that noise is added to the recognition target region 610b. The region in the recognition target region 610b to which noise is added is called a noise-added region 611. The position of the noise-added region 611 is represented by DP(k ｄｐ , l ｄｐ ) and the size is expressed as width W ｄｐ and height H ｄｐ It is expressed as:

[0057] In addition, the position P (w ｐ , h ｐ ) is the upper left position in FIG. 6A in the recognition target area 610b, and the position DP(k ｄｐ , l ｄｐ ) is located in the upper right corner of the noise-added region 611 in FIG. 6A.

[0058] The image recognition unit 120 executes image recognition processing to identify the positions and sizes of the recognition target regions 610a-610c within the captured image 600a. For example, the image recognition unit 120 stores in advance the features of the stored item 210 in the image, and performs a known edge detection process on the captured image 600a using the features of the stored item 210 in the image to identify the positions and sizes of the recognition target regions 610a-610c. Alternatively, for example, the image recognition unit 120 performs a known pattern matching process on the captured image 600a using a pattern image of the stored item 210 to identify the positions and sizes of the recognition target regions 610a-610c.

[0059] The image noise determination unit 122 determines whether or not image noise exists in the recognition target areas 610a to 610c based on changes in luminance distribution, etc. If the luminance distribution differs from the luminance distribution under normal circumstances (when there is no image noise), the image noise determination unit 122 determines that image noise exists, i.e., that noise has been added to the image.

[0060] In the example shown in FIG. 6A, the image noise determination unit 122 determines that noise has been added to the recognition target region 610b, and ｄｐ , l ｄｐ ) and size (width W ｄｐ and height H ｄｐ ) is calculated.

[0061] 6B is a diagram showing an example of a method by which the image noise determination unit 122 calculates the horizontal position and size (width) of the noise-added region 611. The horizontal direction refers to the x-axis direction shown in FIG.

[0062] 6B is a graph showing the horizontal distribution of brightness in the recognition target area 610b. The horizontal axis of FIG. 6B is the position P(w ｐ , h ｐ ) is the origin of the horizontal coordinate k (0 to W ｐ ) indicates the horizontal position within the recognition target area 610b. The vertical axis of Fig. 6B indicates the sum of the vertical luminance values ​​at coordinate k within the recognition target area 610b. The vertical direction is the z-axis direction shown in Fig. 4.

[0063] In Figure 6B, a solid line shows an example of the luminance distribution when noise is added to the recognition target area 610b, and a dotted line shows an example of the luminance distribution when no noise is added to the recognition target area 610b (normal luminance distribution).

[0064] The normal brightness level varies according to the coordinate k as shown by the dotted line in Fig. 6B due to, for example, the influence of the individual information written on the stored item 210. That is, when noise is not added to the recognition target area 610b, the brightness of the recognition target area 610b changes in size as the horizontal coordinate k changes, and changes so as to cross an arbitrarily predetermined brightness threshold Th (repeatedly becoming larger and smaller than the threshold Th).

[0065] When noise is added to the recognition target area 610b, the brightness of the recognition target area 610b has a different distribution from that in normal cases, that is, when no noise is added to the recognition target area 610b.

[0066] The image noise determination unit 122 determines whether noise has been added to the recognition target area 610b based on the difference in the brightness distribution of the recognition target area 610b from normal, and if it determines that noise has been added, calculates the position and size (width and height) of the noise-added area 611.

[0067] The image noise determination unit 122 determines the range of coordinates k that indicates a distribution that is different from the normal distribution of luminance, for example, based on the number of times the distribution of luminance crosses the threshold value Th, and determines the position k that indicates this range. ｄｐ and size (width) W ｄｐ The image noise determination unit 122 calculates the position k ｄｐ and size W ｄｐ is calculated as the horizontal position and size (width) of the noise-added region 611.

[0068] The image noise determination unit 122 stores the luminance distribution of the recognition target area 610b or the characteristics of this luminance distribution (for example, the number of times the luminance distribution crosses the threshold value Th) under normal circumstances, i.e., when no noise is added to the recognition target area 610b.

[0069] The image noise determination unit 122 performs the same process in the vertical direction of the noise addition area 611, and calculates the vertical position l of the noise addition area 611. ｄｐ and size (height) H ｄｐ The vertical direction is the z-axis direction shown in FIG.

[0070] In this way, the image noise determination unit 122 determines the position DP(k ｄｐ , l ｄｐ ) and size (width W ｄｐ and height H ｄｐ ) is calculated.

[0071] The method for calculating the position and size of the noise-added region 611 described above is merely an example. The image noise determination unit 122 may determine whether noise has been added to the recognition target region 610b and calculate the position and size of the noise-added region 611 using an image processing method different from that described above.

[0072] In addition, the way of expressing the position of the recognition target region 610b (in the above example, P(w ｐ , h ｐ )) and how to express the position of the noise-added region 611 (in the above example, DP(k ｄｐ , l ｄｐThe image noise determination unit 122 may express the positions of the recognition target region 610b and the noise-added region 611 in another manner as long as the positions of these regions are uniquely determined within the captured image 600a.

[0073] Returning to the description of the flowchart shown in FIG.

[0074] In S503, if the image noise determination unit 122 detects image noise in the recognition target area 610 (if it determines that noise has been added to the recognition target area 610), the process branches to S504 and proceeds to S505.

[0075] In S505, the sequence control unit 110 changes the position and orientation of the image sensor 20. For example, the sequence control unit 110 moves the image sensor 20 by a predetermined distance in a predetermined direction. The movement direction of the image sensor 20 is, for example, one or both of the x-axis direction (horizontal direction) and the z-axis direction (vertical direction). The movement distance of the image sensor 20 can be determined, for example, depending on the distance between the glass door 201 (transparent body of the storage location) of the refrigerator 200 and the stored item 210. Note that the position and orientation of the image sensor 20 are changed so as not to exceed a predetermined range in order to maintain the angle of view and quality of the captured image 600a required for image recognition and to ensure the performance of the image recognition process.

[0076] In S505, after the position and orientation of the image sensor 20 are changed, the image sensor 20 captures an image of the stored item 210 (a second image of the stored item 210), the image acquisition unit 121 acquires the image captured by the image sensor 20, the image recognition unit 120 performs image recognition processing on the image acquired from the image acquisition unit 121 to identify the recognition target area 610, and the image noise determination unit 122 recalculates and finds the noise-added area 611 in the recognition target area 610. Then, the avoidance method calculation unit 123 calculates the position of the noise-added area 611, i.e., the change in the position of the image noise.

[0077] In S506, based on the change in the position and orientation of the image sensor 20 due to the processing of S505 and the change in the position of the noise-added region 611 determined in the processing of S505, the avoidance method calculation unit 123 calculates a method for changing the position and orientation of the image sensor 20 (the direction and amount of change) to avoid image noise in the recognition target region 610. The position and orientation of the image sensor 20 are changed so as not to exceed a preset range in order to maintain the angle of view and quality of the captured image 600 required for image recognition and to guarantee the performance of the image recognition processing.

[0078] In the image processing device 1 according to this embodiment, image noise in the recognition target area 610 can be avoided (the influence of image noise can be reduced) by changing the position and orientation of the image sensor 20 .

[0079] In S507 , the sequence control unit 110 changes the position and orientation of the image sensor 20 in accordance with the method of changing the position and orientation of the image sensor 20 (the direction and amount of change) calculated by the avoidance method calculation unit 123 .

[0080] Here, the details of the processes in S505 to S507 will be explained with reference to FIGS. 7, 8A, and 8B.

[0081] Figure 7 is a top view showing the positional relationship between the image sensor 20 and the stored items 210a to 210c, which are the objects to be recognized, when the avoidance method calculation unit 123 calculates the method for changing the position and attitude of the image sensor 20 (the direction and amount of change) in this embodiment.

[0082] In this embodiment, an example will be described in which the image sensor 20 is moved by −Δx in the x-axis direction of the coordinate system 400 as shown in FIG. 7 in S505 of FIG. 5, thereby changing the position of the image sensor 20.

[0083] 8A, the procedure in which the avoidance method calculation unit 123 calculates the method of changing the position and orientation of the image sensor 20 (the direction and amount of change) in S506 of FIG. 5 will be described.

[0084] Fig. 8A is a diagram showing an example of an image captured by the image sensor 20 of the image processing device 1 according to this embodiment, and shows an example of a captured image 600b acquired when the position of the image sensor 20 is changed as shown in Fig. 7. Stored items 210a to 210c are captured in the captured image 600b.

[0085] In the captured image 600b, the position of the recognition target area 610b changes from position P to position P' due to the change in the position of the image sensor 20 by -Δx in S505 of FIG. 5. This change in position causes the recognition target area 610b to move by Δw in the x-axis direction (horizontal direction). This is because the entire angle of view moves by Δw as the image sensor 20 moves. Although not explicitly shown in FIG. 8A, the positions of the recognition target areas 610a and 610c have also been changed to positions that have moved by Δw.

[0086] On the other hand, the position of the noise-added region 611 in the captured image 600b changes from position DP to position DP' on the coordinate system based on position P' of the recognition target region 610b because the position of the image sensor 20 changed by -Δx in S505. Due to this change in position, the noise-added region 611 moves by -Δk in the x-axis direction (horizontal direction).

[0087] The position of the noise-added region 611 changes in proportion to the amount of change in the position of the image sensor 20. Therefore, the avoidance method calculation unit 123 can calculate a method for changing the position of the image sensor 20 (the direction and amount of change) to avoid image noise (reduce the influence of image noise) in the recognition target region 610, from the amount of change in the position of the image sensor 20, −Δx, and the amount of change in the position of the noise-added region 611, −Δk. For example, the avoidance method calculation unit 123 calculates a method for changing the position of the image sensor 20 so that the noise-added region 611 does not overlap the recognition target region 610b in the captured image 600b.

[0088] For example, in this embodiment, in S506 of FIG. 5, the avoidance method calculation unit 123 calculates the method (change direction and change amount) of changing the position of the image sensor 20 to avoid image noise in the recognition target area 610 by calculating −k ｄｐThen, in step S507, the sequence control unit 110 calculates that the image sensor 20 should be moved by −k in the x-axis direction. ｄｐ By changing the position of the image sensor 20 by moving it by ×(Δx / Δk), the noise-added region 611 does not overlap the recognition target region 610, and image noise can be avoided in the recognition target region 610.

[0089] Fig. 8B is a diagram showing an example of an image captured by the image sensor 20 of the image processing device 1 according to this embodiment, and shows an example of a captured image 600c acquired when the position of the image sensor 20 is changed in S507 of Fig. 5. That is, the captured image 600c is an example of an image acquired after the position of the image sensor 20 is changed so as to avoid image noise in the recognition target area 610b.

[0090] In the captured image 600c, the recognition target region 610b has moved to position P'', and the noise-added region 611 has moved to position DP'', which is approximately the same position as position P''. Therefore, in the captured image 600c, the noise-added region 611 does not overlap the recognition target region 610b, and the image noise that was added to the recognition target region 610b in FIG. 8A has been avoided. (Position P'' is the upper left position of the recognition target region 610b in FIG. 8B, and position DP'' is the upper right position of the noise-added region 611 in FIG. 8B. Therefore, even if positions P'' and DP'' are in the same position, the noise-added region 611 does not overlap the recognition target region 610b.) Note that, although an example of changing the position of the image sensor 20 has been described in this embodiment, the avoidance method calculation unit 123 can also calculate a change method for the attitude of the image sensor 20 in the same way as for the position. In other words, the avoidance method calculation unit 123 can calculate a method for changing the attitude of the image sensor 20 to avoid image noise in the recognition target area 610 based on the relationship between the amount of change in attitude of the image sensor 20 and the amount of change in position of the noise-added area 611.

[0091] Furthermore, with the image processing device 1 according to this embodiment, for example, with the aim of removing the image noise itself, it is possible to determine whether the cause of the image noise is a foreign substance attached to the glass door 201 or a foreign substance reflected in the glass door 201. For example, the avoidance method calculation unit 123 can determine whether the cause of the image noise is a foreign substance attached to the glass door 201 or a foreign substance reflected in the glass door 201 from a change in the position of the image sensor 20, a change in the position of the recognition target area 610 in the captured image 600, and a change in the position of the noise-added area 611 in the recognition target area 610.

[0092] 5 , after the position of image sensor 20 is changed and the position of recognition target area 610 changes, if the position of noise-added area 611, which is based on changed recognition target area 610, moves in the direction opposite to the direction of change in the position of image sensor 20 and the amount of this movement is greater than a predetermined threshold, avoidance method calculation unit 123 can estimate that the cause of the image noise is a foreign substance (e.g., dirt or fogging) attached to glass door 201. This threshold can be determined, for example, based on the distance between glass door 201 (a transparent body in a storage location) of refrigerator 200 and image sensor 20.

[0093] Furthermore, for example, after the position of the image sensor 20 is changed in S505 of FIG. 5 and the position of the recognition target area 610 changes, if the position of the noise-added area 611 based on the changed recognition target area 610 moves in the same direction as the change in the position of the image sensor 20 and the amount of this movement is equal to or less than the above-mentioned threshold, the avoidance method calculation unit 123 can estimate that the cause of the image noise is a foreign object (for example, light or a device or person in the examination room) reflected on the glass door 201.

[0094] Furthermore, for example, after the position of the image sensor 20 is changed in S505 of Figure 5 and the position of the recognition target area 610 changes, if the amount of movement of the position of the noise addition area 611 based on the changed position of the recognition target area 610 is greater than the above-mentioned threshold, the avoidance method calculation unit 123 can estimate that the cause of the image noise is a foreign object attached to the glass door 201, and if the amount of movement is equal to or less than the above-mentioned threshold, the avoidance method calculation unit 123 can estimate that the cause of the image noise is a foreign object reflected in the glass door 201.

[0095] Returning to the description of the flowchart shown in FIG.

[0096] In S508, the image recognition unit 120 performs a known image recognition process on the image acquired by the image acquisition unit 121 to acquire the individual information of the stored item 210 written on the stored item 210.

[0097] In S503, if the image noise determination unit 122 does not detect image noise in the recognition target area 610 (if it determines that noise has not been added to the recognition target area 610), the process branches to S504 and proceeds to S508.

[0098] Returning to the description of FIG.

[0099] As described above, in the process of S302 in FIG. 3, the image processing device 1 executes the image recognition process to acquire the individual information of the stored items 210a to 210c.

[0100] In S303, the sequence control unit 110 determines whether the image recognition process has been successful for all of the stored items 210a to 210c. If the image recognition process has been successful for all of the stored items 210a to 210c, the process proceeds to S305. If the image recognition process has not been successful, the process proceeds to S304. If the image recognition process has been successful for all of the stored items 210a to 210c, the image processing device 1 can obtain individual information about the stored items 210a to 210c and can recognize and confirm the stored items 210a to 210c inside the refrigerator 200.

[0101] In S304, if the number of times the image recognition process for the stored items 210a to 210c was performed in S302 exceeds a predetermined upper limit, the sequence control unit 110 proceeds to processing S305, and if the number of times the image recognition process for the stored items 210a to 210c was performed does not exceed the upper limit, the sequence control unit 110 proceeds to processing S302 and performs the image recognition process again.

[0102] In S305, sequence control unit 110 determines whether image recognition processing has been performed on all of the recognition objects, that is, stored items 210a to 210e in refrigerator 200. If image recognition processing has been performed on all of the recognition objects (storage items 210a to 210e), the processing shown in the flowchart in Fig. 3 ends. If image recognition processing has not been performed on all of the recognition objects, the processing returns to S301, and image recognition processing is performed on the recognition objects for which image recognition processing has not been performed (in this embodiment, stored items 210d to 210e in refrigerator 200).

[0103] 9 is a diagram showing an example of a display screen 900 output by the display unit 132 in the image processing apparatus 1 according to this embodiment. The display screen 900 includes a setting input screen 901 and a result display screen 902.

[0104] The setting input screen 901 displays information required for processing by the image processing device 1 as it is input by the input unit 131. The information required for processing by the image processing device 1 includes, for example, parameters required for the image recognition unit 120 to specify the recognition target region 610, parameters required for the image noise determination unit 122 to specify the noise addition region 611, and parameters required for the avoidance method calculation unit 123 to specify the procedure for changing the position and attitude of the image sensor 20 and to limit the amount of change.

[0105] The parameters required for the image recognition unit 120 to specify the recognition target area 610 are information indicating the position of characters or symbols to be recognized as individual information of the stored item 210, such as information indicating where on the stored item 210 the individual information is written. The parameters required for the image noise determination unit 122 to specify the noise-added area 611 are, for example, the waveform of the luminance distribution when no noise is added to the recognition target area 610 (normal luminance distribution) and the luminance threshold Th. The parameters required for the avoidance method calculation unit 123 to specify the procedure for changing the position and orientation of the image sensor 20 and to limit the amount of change are, for example, parameters indicating whether to move the image sensor 20 in the x-axis direction or the z-axis direction in S505 and S507 of FIG. 5 , parameters indicating which direction to move first if moving in both the x-axis direction and the z-axis direction, and parameters indicating the maximum movement distance.

[0106] The result display screen 902 displays the processing results of the image processing device 1. For example, the result display screen 902 displays the recognition target area 610 before and after the change in the position and orientation of the image sensor 20 calculated by the image recognition unit 120, the noise addition area 611 before and after the change in the position and orientation of the image sensor 20 calculated by the image noise determination unit 122, and the change method (change direction and change amount) of the position and orientation of the image sensor 20 calculated by the avoidance method calculation unit 123, together with the captured image 600 acquired by the image acquisition unit 121.

[0107] Furthermore, the result display screen 902 may display the cause of the image noise calculated by the avoidance method calculation unit 123. If the cause of the image noise is a foreign object attached to the glass door 201, the result display screen 902 may output an alarm to prompt the user to inspect and clean the glass door 201.

[0108] The image processing device 1 according to the present embodiment has the above-described configuration and can reduce (avoid) the influence of noise added to the recognition target area 610 of the captured image 600. For example, even when recognizing each of the stored items 210 with the glass door 201 of the refrigerator 200 sandwiched between them, the image processing device 1 according to the present embodiment can move the image noise added to the captured image 600 due to the glass door 201 (image noise due to foreign matter attached to the glass door 201 or foreign matter reflected in the glass door 201) from the recognition target area 610, thereby reducing (avoiding) the influence of the noise. Furthermore, the image processing device 1 according to the present embodiment can improve the recognition rate of the recognition target object while maintaining the angle of view and quality of the captured image 600 required for image recognition.

[0109] Second Embodiment An image processing apparatus 1 according to a second embodiment of the present invention will now be described.

[0110] Fig. 10 is a flowchart illustrating the steps of the image recognition process executed by the image processing device 1 according to this embodiment. The flowchart shown in Fig. 10 corresponds to the flowchart shown in Fig. 5 in the first embodiment, and is a flowchart illustrating the process of S302 in Fig. 3 (the process of executing the image recognition process to acquire individual information of the stored item 210).

[0111] The flowchart shown in Fig. 10 includes the process of S1001 added to the flowchart shown in Fig. 5. S1001 is a process executed after S507.

[0112] In S1001, the avoidance method calculation unit 123 of the control device 100 stores the method of changing the position and attitude of the image sensor 20 (change direction and change amount) calculated in S506 as teaching data for the sequence control unit 110.

[0113] Furthermore, in S1001 , the avoidance method calculation unit 123 can also store the position and orientation of the image sensor 20 changed by the sequence control unit 110 in S507 as teaching data for the sequence control unit 110 .

[0114] The avoidance method calculation unit 123 can use this stored data, for example, when the sequence control unit 110 changes the position and orientation of the image sensor 20 in S505.

[0115] The image processing device 1 according to this embodiment has the above configuration, and when the same operation sequence for image recognition is repeatedly performed for each image recognition process, and when image noise is always present at a certain position and range in the captured image, the execution of the series of processes (S505 to S1001) for avoiding image noise can be omitted after the avoidance method calculation unit 123 stores the method for changing the position and orientation of the image sensor 20 (and the position and orientation after the change). Therefore, the image processing device 1 according to this embodiment can reduce the time required to acquire the individual information of the stored item 210 described on the stored item 210 by performing image recognition processing on the acquired image.

[0116] In addition to the above, the image processing apparatus 1 according to this embodiment also provides the effects described in the first embodiment.

[0117] An image processing device 1 according to a third embodiment of the present invention will be described. The image processing device 1 according to this embodiment determines whether there is a change in image noise over time. If there is a change in image noise over time, the image processing device 1 waits until there is no change in the image noise over time or until the range of the image noise becomes small, and then detects the image noise and calculates a method for changing the position and orientation of the image sensor 20 to avoid the image noise.

[0118] Fig. 11 is a flowchart illustrating the steps of the image recognition process executed by the image processing device 1 according to this embodiment. The flowchart shown in Fig. 11 corresponds to the flowchart shown in Fig. 5 in the first embodiment, and explains the process of S302 in Fig. 3 (the process of executing the image recognition process to acquire individual information of the stored item 210).

[0119] The flowchart shown in Fig. 11 includes the additional steps of S1101 and S1102 in the flowchart shown in Fig. 5. S1101 and S1102 are executed after S504.

[0120] In S503, if the image noise determination unit 122 detects image noise in the recognition target area 610 (if it determines that noise has been added to the recognition target area 610), the process branches to S504 and proceeds to S1101.

[0121] In S1101, the image noise determination unit 122 determines whether the image noise detected in S503 has changed over time. Specifically, the image noise determination unit 122 determines whether the luminance distribution and range of the noise-added region 611 ( FIG. 6A ) detected in S503 have changed over time. For example, the image noise determination unit 122 determines whether the luminance distribution and range of the noise-added region 611 have changed over time by calculating the difference between a plurality of captured images captured at different times.

[0122] For example, if the amount of change over time in the image noise (the amount of change over time in the brightness distribution and range of the noise-added area 611) becomes equal to or less than a predetermined value within a predetermined period of time, the image noise determination unit 122 determines that there is no change over time in the image noise.

[0123] If there is a change in the image noise over time, the image noise determination unit 122 repeats the process of S1101, that is, the determination of whether there is a change in the image noise over time.

[0124] If there is no change in the image noise over time, the process proceeds to step S1102. Even if there is a change in the image noise over time, the process may proceed to step S1102 if the size (range) of the noise-added region 611 becomes smaller than a predetermined size.

[0125] In S1102, the image noise determination unit 122 again determines whether or not image noise has been detected in the recognition target area 610. If image noise has been detected in the recognition target area 610, the process proceeds to S505, and the image processing device 1 executes a series of processes (S505 to S507) for avoiding image noise. If image noise has not been detected in the recognition target area 610, the process proceeds to S508, and the image processing device 1 executes image recognition processing on the acquired image to acquire the individual information of the stored item 210 described on the stored item 210.

[0126] Note that if the image noise no longer changes over time after it has been determined in the process of S1101 that there is a change over time, and if the process of S505 estimates that the cause of the image noise is a foreign object (e.g., a person or a moving object in the examination room) reflected in the glass door 201, the result display screen 902 of the display screen 900 ( FIG. 9 ) may display an alarm. This alarm is intended to prevent people or moving objects in the examination room from being reflected in the glass door 201, and may, for example, prompt people in the examination room to move away from the mobile cart 10.

[0127] The image processing device 1 according to this embodiment has the above configuration, and when there is a change in image noise over time, it waits a predetermined time until the change in image noise over time disappears or until the range of the image noise (the size of the noise-added region 611) becomes smaller than a predetermined size, and then detects the image noise. Then, it calculates a method for changing the position and orientation of the image sensor 20 to avoid the image noise.

[0128] For example, if there is a short-term movement (change) in a person (background image) reflected in the glass door 201 of the refrigerator 200, the image processing device 1 according to this embodiment does not execute the series of processes (S505 to S507) for avoiding image noise every time, but waits until the person stops moving before detecting the image noise. This prevents the method of changing the position and attitude of the image sensor 20 (the direction and amount of change) from exceeding a predetermined maximum movement distance. In other words, the image processing device 1 according to this embodiment can improve the success rate of reducing (avoiding) the effects of noise when there is a change in image noise over time and when there is a short-term movement (change) in a foreign object reflected in the glass door 201.

[0129] Furthermore, in the image processing device 1 according to this embodiment, when the glass door 201 of the refrigerator 200 is fogged due to condensation or the like, i.e., when the glass door 201 is fogged, the disappearance of this fog can be detected as a change in image noise over time. In the image processing device 1 according to this embodiment, image recognition processing can be performed after the fog on the glass door 201 has disappeared or after the size of the fog has become sufficiently small, thereby avoiding image noise due to the fog on the glass door 201. Furthermore, in the image processing device 1 according to this embodiment, it is possible to prevent the method of changing the position and attitude of the image sensor 20 (the direction and amount of change) from exceeding a predetermined maximum movement distance while the glass door 201 is fogged, thereby improving the success rate of reducing (avoiding) the effects of noise.

[0130] In addition to the above, the image processing apparatus 1 according to this embodiment also provides the effects described in the first embodiment.

[0131] An image processing device 1 according to a fourth embodiment of the present invention will be described.

[0132] 12 is a flowchart showing a procedure executed by the image processing device 1 according to this embodiment to acquire individual information of the stored item 210 in the refrigerator 200. The flowchart shown in FIG. 12 corresponds to the flowchart shown in FIG. 3 in the first embodiment.

[0133] The flowchart shown in Fig. 12 includes steps S1201 to S1204 in addition to the steps S1201 to S1204 in the flowchart shown in Fig. 3. Steps S1201 to S1204 are executed after step S305.

[0134] In S305, if the sequence control unit 110 determines that the image recognition process has been performed on all of the stored items 210 inside the refrigerator 200, which are the objects to be recognized, the process proceeds to S1201.

[0135] In S1201, the sequence control unit 110 determines whether the image recognition process has been successful for all stored items 210 inside the refrigerator 200, which are the objects to be recognized. If information that should be obtained as individual information of the stored items 210 (for example, information for identifying the stored items 210 and information about the attributes of the stored items 210) has been obtained, the sequence control unit 110 determines that acquisition of the individual information has been successful and that the image recognition process has been successful. If the image recognition process has been successful for all stored items 210, the process shown in the flowchart in Fig. 12 ends. If there is a stored item 210 for which the image recognition process has failed, the process proceeds to S1202.

[0136] In S1202, the sequence control unit 110 determines whether the time required for the image recognition process executed on the stored item 210 for which the image recognition process failed (if the image recognition process was executed multiple times on the stored item 210, the total time required for all of the multiple image recognition processes) is within a predetermined specified time. If the time required for the image recognition process is within the specified time, the process proceeds to S1203. If the time required for the image recognition process exceeds the specified time, the process proceeds to S1204.

[0137] For example, the sequence control unit 110 may include a timer and use the timer to determine the time required for the image recognition process. Alternatively, the sequence control unit 110 may include a counter and use the counter to count the number of times the image recognition process has been performed, and estimate the time required for the image recognition process from the count.

[0138] In S1203, the result display screen 902 of the display screen 900 (FIG. 9) displays a message indicating that it is permissible to temporarily open the glass door 201 of the refrigerator 200. Upon seeing this message, the laboratory technician opens the glass door 201 temporarily.

[0139] After this, the image processing device 1 returns to process 301 and performs image recognition processing again on the stored item 210 for which the image recognition processing failed, with the glass door 201 open. Because the glass door 201 is open, the image sensor 20 captures an image of the stored item 210 without passing through the glass door 201. The image processing device 1 then performs image recognition processing on this image, thereby avoiding image noise caused by the glass door 201. Note that the laboratory technician closes the temporarily opened glass door 201 after a predetermined time has elapsed. The time for which the glass door 201 is open is set to the time allowed for the stored item 210 in terms of temperature control.

[0140] In S1204, the image processing device 1 suspends the image recognition process and moves the mobile cart 10 from in front of the refrigerator 200. As a result, the image recognition unit 120 suspends the process of acquiring individual information of the stored item 210 by performing the image recognition process on the image captured by the image sensor 20.

[0141] The image processing device 1 according to this embodiment has the above configuration, and if the image recognition process fails, the glass door 201 is temporarily opened and the image recognition process is executed, thereby increasing the probability that the image recognition process will be successful for all stored items 210. Furthermore, for example, when the range of influence of image noise is large and it is difficult to avoid the image noise, the configuration provided in the image processing device 1 according to this embodiment is effective.

[0142] If opening of the glass door 201 is not permitted, the image processing device 1 executes the process of S1204 and interrupts the image recognition process. Even in such a case, the mobile cart 10 immediately moves from in front of the refrigerator 200, so that the work of the laboratory technician can be affected as little as possible.

[0143] In addition to the above, the image processing apparatus 1 according to this embodiment also provides the effects described in the first embodiment.

[0144] It should be noted that the present invention is not limited to the above-described embodiments, and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to embodiments that include all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to delete part of the configuration of each embodiment, or to add or replace other configurations.

[0145] 1...image processing device, 10...mobile cart, 11...driving mechanism, 12...terminal driving mechanism, 20...image sensor, 100...control device, 110...sequence control unit, 111...operation control unit, 120...image recognition unit, 121...image acquisition unit, 122...image noise determination unit, 123...avoidance method calculation unit, 130...input / output processing unit, 131...input unit, 132...display unit, 200...refrigerator, 201...glass door, 210, 210a to 210e...stored item, 400...coordinate system, 600, 600a to 600c...captured image, 610, 610a to 610c...recognition target area, 611...noise addition area, 900...display screen, 901...setting input screen, 902...result display screen.

Claims

1. An image sensor that captures an image of an object to be recognized; a drive mechanism in which the image sensor is installed and which can change the position and attitude of the image sensor; a control unit that controls the drive mechanism to change the position and attitude of the image sensor; an image recognition unit that performs image recognition processing on the image captured by the image sensor to identify a recognition target area which is an area in the image from which information is acquired; an image noise determination unit that determines the presence or absence of image noise in the recognition target area based on changes in luminance distribution, and if the image noise is detected, calculates the position and size of a noise-added area in the recognition target area where the image noise is added; and an avoidance method calculation unit that calculates a method for changing the position and attitude of the image sensor to reduce the effect of the image noise; wherein if the image noise determination unit detects the image noise in the recognition target area in the first image captured by the image sensor, the control unit changes the position and attitude of the image sensor, and the image sensor whose position and attitude have been changed captures a second image, and the image recognition unit identifies the recognition target area in the second image, the image noise determination unit calculates a position and a size of the noise-added region in the recognition target region in the second image; the avoidance method calculation unit calculates the modification method based on a change in position and orientation of the image sensor and a change in position of the noise-added region in the second image from the first image; and the control unit modifies the position and orientation of the image sensor in accordance with the modification method calculated by the avoidance method calculation unit.

2. The image processing device according to claim 1, wherein the image sensor captures the image through a transparent body, and the avoidance method calculation unit determines whether the cause of the image noise is a foreign object attached to the transparent body or a foreign object reflected in the transparent body based on changes in the position and orientation of the image sensor and changes in the position of the noise-added area in the recognition target area.

3. The image processing device according to claim 1, wherein the avoidance method calculation unit stores the calculated change method.

4. The image processing device according to claim 1, wherein the avoidance method calculation unit stores the position and orientation of the image sensor changed by the control unit.

5. The image processing device according to claim 1, wherein the image noise determination unit determines whether or not there is a change in the image noise over time, and the avoidance method calculation unit, if there is a change in the image noise over time, calculates the change method when the image noise no longer changes over time.

6. The image processing device according to claim 1, wherein the image noise determination unit determines whether or not there is a change in the image noise over time, and the avoidance method calculation unit calculates the change method when the size of the noise-added area becomes smaller than a predetermined size if there is a change in the image noise over time.

7. An image processing device according to claim 1, comprising: a display unit that outputs a display screen; a plurality of the recognition objects are stored in a storage location having a door at least partially made of a transparent body; the image sensor captures the image through the transparent body; the image recognition unit performs the image recognition process on the image to obtain information about the recognition object; if there is a recognition object for which the image recognition process has failed and the time required for the image recognition process is within a predetermined specified time, the display screen displays a message that the door may be opened; the image sensor captures the image without passing through the transparent body while the door is open; and if there is a recognition object for which the image recognition process has failed and the time required for the image recognition process exceeds the specified time, the image recognition unit interrupts the image recognition process.

Images