Image Processing Apparatus, Image Processing Program, and Image Processing Method

Abstract

The image processing device obtains an image depicting a subject taking a predetermined posture and a predetermined shape, a first reference image, and a second reference image, wherein the first reference image depicts a subject taking a first posture whose relationship with the predetermined posture is known and a first shape whose relationship with the predetermined shape is known, the second reference image depicts a subject taking at least one of a second posture different from the predetermined posture and a second shape different from the predetermined shape, and extracts feature data representing the features of the subject depicted in the image, extracts first reference data representing the features of the subject depicted in the first reference image, extracts second reference data representing the features of the subject depicted in the second reference image, and uses the first reference data to determine the part where the feature represented by the feature data is located in the first reference image, and determines the part where the feature represented by the feature data is located in the subject taking at least one of the second posture and the second shape based on at least one of the relationship between the first posture and the second posture and the relationship between the first shape and the second shape.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing program, and an image processing method. Background Art

[0002] In recent years, a technique of using in combination a stereoscopic image that depicts a subject three-dimensionally and an image that depicts the subject has been effectively applied in various fields. For example, Patent Document 1 discloses a technique having the following steps: determining the position of a transmission image of an inspection position in a radiation transmission image; and determining the inspection position in a three-dimensional image based on the position of the transmission image.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: Japanese Patent Laid-Open No. 2019-060808

[0006] However, in the technique of Patent Document 1, a substrate holding unit that holds a substrate as an object to be inspected and a detector are relatively moved with respect to a radiation generator to obtain a plurality of transmission images, and the positions of images of common patterns and marks in all of these transmission images on a reconstructed image are determined to determine an inspection surface image. That is, in the technique of Patent Document 1, if there are no images of common patterns or marks in a plurality of transmission images, the inspection surface image cannot be determined.

[0007] On the other hand, for example, in the field of food processing, due to restrictions related to the layout of a food processing line, production efficiency, etc., it is sometimes required to take an image of a subject and use the image after performing processing such as transporting and cutting the subject using a robotic arm. However, since this processing may change at least one of the posture and shape of the subject, the image may not be usable. Summary of the Invention

[0008] Problems to be Solved by the Invention

[0009] Therefore, an object of the present invention is to provide an image processing apparatus, an image processing program, and an image processing method that can use an image taken before at least one of the posture and shape of a subject is changed even after at least one of the posture and shape of the subject is changed.

[0010] Means for Solving the Problems

[0011] (1) To solve the above problems, an image processing apparatus according to one embodiment of the present invention includes: an acquisition unit that acquires an image depicting a subject having a predetermined posture and a predetermined shape, a first reference image, and a second reference image, where the first reference image depicts the subject having a first posture whose relationship with the predetermined posture is known and a first shape whose relationship with the predetermined shape is known, and the second reference image depicts the subject having at least one of a second posture different from the predetermined posture and a second shape different from the predetermined shape; and a determination unit that extracts feature data representing the features of the subject depicted in the image, extracts first reference data representing the features of the subject depicted in the first reference image, extracts second reference data representing the features of the subject depicted in the second reference image, and uses the first reference data to determine the location of the feature represented by the feature data in the first reference image, and uses at least one of the relationship between the first posture of the subject derived from the first reference data and the second posture of the subject derived from the second reference data, and the relationship between the first shape of the subject derived from the first reference data and the second shape of the subject derived from the second reference data to determine the location of the feature represented by the feature data in the subject having at least one of the second posture and the second shape.

[0012] (2) In the image processing apparatus according to the above (1) embodiment, the determination unit may also perform at least one of the process of using the first reference data to determine the first posture and the process of using the second reference data to determine the second posture.

[0013] (3) In the image processing apparatus according to the above (2) embodiment, the determination unit extracts at least one of the first reference data and the second reference data, where the first reference data represents a feature whose position in the subject remains fixed even when a predetermined process is performed on the subject, and the second reference data represents a feature whose position in the subject remains fixed even when the predetermined process is performed on the subject.

[0014] (4) In the image processing apparatus according to any one of the above (2) or (3) embodiments, the determination unit performs at least one of the process of extracting the first reference data and the process of extracting the second reference data, where the first reference data represents two first feature points located on a first straight line and the first feature point different from the points located on the first straight line, and the second reference data represents two second feature points located on a second straight line and the second feature point different from the points located on the second straight line.

[0015] (5) To solve the above problems, an image processing program according to an aspect of the present invention causes a computer to implement an acquisition function and a determination function. The acquisition function acquires an image of a photographed object taking a predetermined posture and a predetermined shape, a first reference image, and a second reference image. The first reference image depicts the photographed object taking a first posture whose relationship with the predetermined posture is known and a first shape whose relationship with the predetermined shape is known. The second reference image depicts the photographed object taking at least one of a second posture different from the predetermined posture and a second shape different from the predetermined shape. The determination function extracts feature data representing the features of the photographed object depicted in the image, extracts first reference data representing the features of the photographed object depicted in the first reference image, extracts second reference data representing the features of the photographed object depicted in the second reference image, and uses the first reference data to determine the position of the feature represented by the feature data in the first reference image. Using at least one of the relationship between the first posture of the photographed object derived from the first reference data and the second posture of the photographed object derived from the second reference data, and the relationship between the first shape of the photographed object derived from the first reference data and the second shape of the photographed object derived from the second reference data, it is determined the position of the feature represented by the feature data in the photographed object taking at least one of the second posture and the second shape.

[0016] (6) To solve the above problems, an image processing method according to an aspect of the present invention includes an acquisition step and a determination step. In the acquisition step, an image depicting a subject having a predetermined posture and a predetermined shape, a first reference image, and a second reference image are acquired. The first reference image depicts the subject having a first posture whose relationship with the predetermined posture is known and a first shape whose relationship with the predetermined shape is known. The second reference image depicts the subject having at least one of a second posture different from the predetermined posture and a second shape different from the predetermined shape. In the determination step, feature data representing the features of the subject depicted in the image is extracted, first reference data representing the features of the subject depicted in the first reference image is extracted, second reference data representing the features of the subject depicted in the second reference image is extracted, and the position of the feature represented by the feature data in the first reference image is determined using the first reference data. The position of the feature represented by the feature data in the subject having at least one of the second posture and the second shape is determined using at least one of the relationship between the first posture of the subject derived from the first reference data and the second posture of the subject derived from the second reference data, and the relationship between the first shape of the subject derived from the first reference data and the second shape of the subject derived from the second reference data.

[0017] Advantages of the Invention

[0018] According to the present invention, even after at least one of the posture and shape of the subject has changed, an image taken before at least one of the posture and shape of the subject has changed can be used. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 FIG. is an example showing a food processing line according to an embodiment of the present invention.

[0020] Figure 2 FIG. is an example showing a functional structure of an image processing apparatus according to an embodiment of the present invention.

[0021] Figure 3 FIG. is an example showing feature points extracted from an X-ray image and first feature points extracted from a first stereoscopic image according to an embodiment of the present invention.

[0022] Figure 4 FIG. is an example showing second feature points extracted from a second stereoscopic image according to an embodiment of the present invention.

[0023] Figure 5It is a diagram showing an example of an object to be photographed according to an embodiment of the present invention and an X-ray image depicting the characteristics of the object to be photographed.

[0024] Figure 6 It is a diagram showing an example of an object to be photographed according to an embodiment of the present invention and a first stereoscopic image depicting the characteristics of the object to be photographed.

[0025] Figure 7 It is a diagram showing an example of an object to be photographed according to an embodiment of the present invention and a second stereoscopic image depicting the characteristics of the object to be photographed.

[0026] Figure 8 It shows in Figure 7 A diagram showing an example of the part where the feature represented by the feature data is located in the object to be photographed shown.

[0027] Figure 9 It is a diagram showing an example of an object to be photographed according to an embodiment of the present invention and a second stereoscopic image depicting the characteristics of the object to be photographed.

[0028] Figure 10 It shows in Figure 9 A diagram showing an example of the part where the feature shown by the feature data is located in the object to be photographed shown.

[0029] Figure 11 It is a flowchart showing an example of the processing executed by the image processing apparatus according to an embodiment of the present invention. Detailed Embodiments

[0030] Refer to Figures 1 to 10 The image processing apparatus according to the embodiment will be described. Figure 1 It is a diagram showing an example of a food processing line according to an embodiment of the present invention. As Figure 1 Shown, the food processing line 1 includes a conveyor belt 11, a conveyor belt 12, a conveyor belt 13, an X-ray imaging device 20, a first stereoscopic image imaging device 21, a second stereoscopic image imaging device 22, an articulated robot 31, and an articulated robot 32. In the following description, the case where the object to be photographed P is a pork leg including the pubic bone, the coccyx, and the hip bone equivalent to a human pelvis will be described as an example.

[0031] The conveyor belts 11, 12, and 13 all rotate the crawlers to convey the object to be photographed P. In addition, the conveyor belts 11, 12, and 13 all stop the crawlers as needed to stop conveying the object to be photographed P. The conveyor belts 11 and 13 are conveyor belts that convey the object to be photographed P straight. On the other hand, the conveyor belt 12 is a conveyor belt that changes the conveying direction of the object to be photographed P by 90 degrees.

[0032] While the X-ray imaging device 20 conveys the object P to be imaged using the conveyor belt 11, it irradiates the object P with X-rays in a state where the posture remains unchanged and captures an X-ray image of the object P. The X-ray imaging device 20 includes an X-ray tube and an X-ray detector. The X-ray image is an example of an image depicting the object P in a predetermined posture and a predetermined shape, and at least depicts the flesh and bones included in the object P. In addition, the X-ray image is stored in a storage medium provided inside or outside the X-ray imaging device 20. The method of using the X-ray image will be described later. In addition, an X-ray shielding device is attached to the X-ray imaging device 20, and this X-ray shielding device is provided to prevent the leakage of X-rays and thus protect the operators or the equipment arranged around.

[0033] The first stereoscopic imaging device 21 is stationary on the conveyor belt 11 and captures a first stereoscopic image that stereoscopically depicts the object P to be imaged, where the object P assumes the same posture and the same shape as the posture when the X-ray image is captured by the X-ray imaging device 20. The first stereoscopic image is an example of a first reference image depicting the object to be imaged, where the object assumes a first posture with a known relationship to the predetermined posture and a first shape with a known relationship to the predetermined shape. In addition, the first stereoscopic image includes, for example, an image in which gradations representing the depth in the X-axis direction are assigned to each pixel, an image in which gradations representing the depth in the Y-axis direction are assigned to each pixel, and an image in which gradations representing the depth in the Z-axis direction are assigned to each pixel. The X-axis, Y-axis, and Z-axis mentioned here are all coordinate axes of a predetermined three-dimensional coordinate system. The first stereoscopic image is stored in a storage medium provided inside or outside the first stereoscopic imaging device 21 and the X-ray imaging device 20. The method of using the first stereoscopic image will be described later.

[0034] After the X-ray image of the object P is captured by the X-ray imaging device 20 and the first stereoscopic image is captured by the first stereoscopic imaging device 21, the object P is conveyed by the conveyor belt 11 and the conveyor belt 12 and thus placed on the conveyor belt 13.

[0035] The posture of the object P placed on the conveyor belt 13 is different from the posture of the object P when the X-ray image is captured by the X-ray imaging device 20 and the first stereoscopic image is captured by the first stereoscopic imaging device 21. This is because the posture of the object P changes due to vibrations during conveyance by the conveyor belt 11, the conveyor belt 12, or the conveyor belt 13, or vibrations when crossing the joint between the conveyor belt 11 and the conveyor belt 12 or the joint between the conveyor belt 12 and the conveyor belt 13.

[0036] The second three-dimensional image capturing device 22 captures a second three-dimensional image that three-dimensionally depicts the object P placed on the conveyor belt 13. In addition, the second three-dimensional image capturing device 22 captures the second three-dimensional image in a state where the processing of at least one of the multi-joint robot 31 and the multi-joint robot 32 is temporarily stopped. The second three-dimensional image is an example of a second reference image depicting the object P, and the object P takes at least one of a second posture different from the above-mentioned predetermined posture and a second shape different from the above-mentioned predetermined shape. In addition, for example, similar to the above-mentioned first three-dimensional image, the second three-dimensional image includes an image in which gradation representing the depth in the X-axis direction is assigned to each pixel, an image in which gradation representing the depth in the Y-axis direction is assigned to each pixel, and an image in which gradation representing the depth in the Z-axis direction is assigned to each pixel. In addition, the second three-dimensional image is captured at a position different from the position where the X-ray image is captured. The usage method of the second three-dimensional image will be described later.

[0037] The multi-joint robot 31 is, for example, a vertically articulated robot, and a hand for gripping the object P is attached to the front end. The multi-joint robot 32 is, for example, a vertically articulated robot, and a knife for cutting the meat contained in the object P is attached to the front end. The multi-joint robot 31 appropriately adjusts the position and posture of the object P by gripping or moving the object P. The multi-joint robot 32 cuts the meat contained in the object P in a state where the position and posture of the object P are adjusted, and removes the tailbone and hip bone from the object P. The multi-joint robot 31 and the multi-joint robot 32 perform these processes, for example, based on the positions of the feature points in the second three-dimensional image determined by the image processing device 100 described later.

[0038] Figure 2 It is a diagram showing an example of the functional structure of the image processing device according to the embodiment of the present invention. As Figure 2 shown, the image processing device 100 includes an acquisition unit 101 and a determination unit 102.

[0039] At least a part of the functions of the image processing apparatus 100, the acquisition unit 101, and the determination unit 102 are implemented by hardware including circuitry executing a software program. The hardware mentioned here is, for example, a CPU (Central Processing Unit), an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit). In addition, the above program is stored in a storage device having a storage medium. The storage medium mentioned here is, for example, an HDD (Hard Disk Drive), a flash memory, a ROM (Read Only Memory), or a DVD (Digital Versatile Disc). In addition, the above program may be a differential program that implements a part of the functions of the image processing apparatus 100.

[0040] The acquisition unit 101 acquires an image depicting the subject P having a predetermined posture and a predetermined shape. In addition, the acquisition unit 101 acquires a first reference image depicting the subject, the subject having a first posture whose relationship with the predetermined posture is known and a first shape whose relationship with the predetermined shape is known. For example, the acquisition unit 101 acquires a first stereoscopic image depicting the subject P having the same posture as the predetermined posture and the same shape as the predetermined shape. In addition, the acquisition unit 101 acquires a second reference image depicting the subject having at least one of a second posture different from the predetermined posture and a second shape different from the predetermined shape. For example, the acquisition unit 101 acquires a second stereoscopic image depicting the subject P having a posture different from the predetermined posture. The predetermined posture mentioned here is, for example, the posture of the subject P in the case of performing X-ray image shooting by the X-ray shooting apparatus 20 and first stereoscopic image shooting by the first stereoscopic image shooting apparatus 21.

[0041] Figure 3FIG. is an example showing feature points extracted from an X-ray image according to an embodiment of the present invention and first feature points extracted from a first stereoscopic image. The determination unit 102 extracts feature data representing features of the object P depicted in the X-ray image. For example, the determination unit 102 extracts feature data representing feature points on the object P depicted in the X-ray image. The feature points mentioned here are, for example, points representing the shape features of the boundary between the bone B and the flesh N of the object P depicted in the X-ray image. For example, the determination unit 102 extracts Figure 3 feature data representing the points F11, F12, F13, F14, and F15 shown as feature points. In addition, the method by which the determination unit 102 extracts feature data from the X-ray image is not particularly limited.

[0042] In addition, the determination unit 102 extracts first reference data representing features of the object P depicted in the first reference image. For example, the determination unit 102 extracts first reference data representing first feature points on the object P depicted in the first stereoscopic image. The first feature points mentioned here are two points located on the first straight line and points different from the points located on the first straight line, and are points located on the surface of the object P. In addition, the first feature points are preferably points whose positional relationship with the feature points in the object P remains fixed even when a predetermined process is performed on the object P, for example, a process performed by the multi-joint robot 31 and the multi-joint robot 32. Examples of such points include points located at the ends of the cross-section of the coccyx and points located at the ends of the cross-section of the pubis. The determination unit 102, for example, extracts Figure 3 first reference data representing the points C11, C12, and C13 shown as first feature points. In addition, the method by which the determination unit 102 extracts first reference data from the first stereoscopic image is not particularly limited.

[0043] Next, the determination unit 102 determines the position of the feature represented by the feature data in the first reference image using the first reference data. For example, the determination unit 102 determines the position of the feature points in the first stereoscopic image using the first feature points. Next, the determination unit 102 Figure 3 uses the point C11 shown as the origin, the coordinate axis from the point C11 toward the point C12 as the first axis A11, and the coordinate axis from the point C11 toward the point C13 as the second axis A12. In addition, the determination unit 102 determines the third axis A13 in such a way that the first axis A11, the second axis A12, and the third axis A13 form a right-handed coordinate system. Then, the determination unit 102 calculates the coordinates of each of the points F11, F12, F13, F14, and F15 in the three-dimensional coordinates determined by the first axis A11, the second axis A12, and the third axis A13.

[0044] The determination unit 102 determines the first pose of the captured object P derived from the first reference data. Specifically, the determination unit 102 uses the first reference data representing Figure 3 the points C11, C12, and C13 shown in FIG. 1, and determines the pose of the captured object P represented by the first stereo image. In addition, the determination unit 102 determines the first shape of the captured object P derived from the first reference data. For example, the determination unit 102 determines the contour of the captured object P detected by performing edge detection on the first reference image as the first shape of the captured object P.

[0045] Figure 4 FIG. 2 is a diagram showing an example of the second feature points extracted from the second stereo image of the embodiment of the present invention. The determination unit 102 extracts the second reference data representing the features of the captured object P depicted in the second reference image. For example, the determination unit 102 extracts the second reference data representing the second feature points on the captured object P depicted in the second stereo image. The second feature points mentioned here are two points located on the second straight line and points different from the points located on the second straight line, and are points located on the surface of the captured object P. In addition, similar to the first feature points, the second feature points are preferably points whose positional relationship with the feature points in the captured object P remains fixed even when a predetermined process is performed on the captured object P, for example, the processes performed by the multi-joint robot 31 and the multi-joint robot 32. The determination unit 102 extracts, for example, the second reference data representing Figure 4 the points C21, C22, and C23 shown in FIG. 2 as the second feature points. In addition, the method by which the determination unit 102 extracts the second reference data from the second stereo image is not particularly limited.

[0046] In addition, the determination unit 102 sets Figure 4 the point C21 shown in FIG. 3 as the origin, the coordinate axis from the point C21 toward the point C22 as the first axis A21, and the coordinate axis from the point C21 toward the point C23 as the second axis A22. In addition, the determination unit 102 determines the third axis A23 such that the first axis A21, the second axis A22, and the third axis A23 form a right-handed coordinate system.

[0047] The determination unit 102 determines the second pose of the captured object P derived from the second reference data. Specifically, the determination unit 102 uses the second reference data representing Figure 4 the points C21, C22, and C23 shown in FIG. 3, and determines the pose of the captured object P represented by the second stereo image. In addition, the determination unit 102 determines the second shape of the captured object P derived from the second reference data. For example, the determination unit 102 determines the contour of the captured object P detected by performing edge detection on the second reference image as the second shape of the captured object P.

[0048] Furthermore, the determination unit 102 determines the part of the subject P in the second posture and / or the second shape where the feature represented by the feature data is located. At this time, the determination unit 102 uses at least one of the relationship between the first posture of the subject P derived from the first reference data and the second posture of the subject P derived from the second stereoscopic image, and the relationship between the first shape of the subject P derived from the first reference data and the second shape of the subject P derived from the second reference data. Additionally, in this case, the part determined by the determination unit 102 may be a position on the second reference image or a position on the X-ray image of the subject in the second posture and / or the second shape.

[0049] For example, the determination unit 102 derives the correspondence between the three-dimensional coordinate system determined by the first axis A11, the second axis A12, and the third axis A13 and the three-dimensional coordinate system determined by the first axis A21, the second axis A22, and the third axis A23. Then, the determination unit 102 uses this correspondence to convert Figure 3 the points F11, F12, F13, F14, and F15 shown into points F21, F22, F23, F24, and F25 on the three-dimensional coordinates determined by the first axis A21, the second axis A22, and the third axis A23 shown by Figure 4 The points F21, F22, F23, F24, and F25 are the reference when the multi-joint robot 31 and the multi-joint robot 32 perform the above processing.

[0050] Next, an example of the feature of the subject P represented by the feature data, the feature of the subject P represented by the first reference data, the feature of the subject P represented by the second reference data, and the part of the feature of the subject P represented by the feature data in the second posture and / or the second shape will be described.

[0051] Figure 5 is a diagram showing an example of a subject of an embodiment of the present invention and an X-ray image depicting the features of the subject. Figure 5 The subject shown includes a bone and the flesh attached around the bone. Figure 5 The points A, J, C, D, E, F, G, H, N8, C5, C4, N7, and C1 shown are all examples of feature points represented by the feature data.

[0052] Figure 6 is a diagram showing an example of a subject of an embodiment of the present invention and a first stereoscopic image depicting the features of the subject. Figure 6 The first stereoscopic image shown is related to Figure 5The subject shown is the same, and it is an image depicting the three-dimensional external shape of a subject that assumes the same pose and the same shape as the subject. Figure 6 The points c1, c4, c5, n7, and n8 shown are each an example of a first feature point represented by first reference data, and are points on the cross-section of a bone on the surface of the subject.

[0053] Figure 7 It is a diagram showing an example of a subject according to an embodiment of the present invention and a second stereoscopic image depicting the features of the subject. Figure 7 The second stereoscopic image shown and Figure 5 and Figure 6 The subject shown are the same, and it is an image depicting the three-dimensional external shape of a subject that assumes a different pose and a different shape from the subject. Figure 7 The points c1, c4, c5, n7, and n8 shown are each an example of a second feature point represented by second reference data, and are points on the cross-section of a bone on the surface of the subject.

[0054] In addition, Figure 7 The points c1, n7, and n8 shown are respectively made to correspond to the points c1, n7, and n8 shown by using feature-based matching of three feature points. And the correspondence of these three points determines the relationship between the pose and shape of the subject depicted in the first stereoscopic image shown in Figure 6 and the pose and shape of the subject depicted in the second stereoscopic image shown in Figure 6 In addition, feature-based matching is a technique for extracting features such as edges and corners from multiple images and performing matching between multiple images based on the relationships of these features. Figure 7 It is a diagram showing an example of the part where the features represented by the feature data are located in the subject shown in

[0055] Figure 8 It is a diagram showing an example of Figure 7 the part where the features represented by the feature data are located in the subject shown. Figure 8 The image shown and Figure 7 the subject shown are the same, and it is an X-ray image depicting a subject that assumes the same pose and the same shape as the subject. Figure 8 The points A, J, C, D, E, F, G, H, N8, C5, C4, N7, and C1 shown respectively correspond to Figure 5 the points A, J, C, D, E, F, G, H, N8, C5, C4, N7, and C1 shown.

[0056] Figure 9This is a diagram showing an example of an object to be photographed in an embodiment of the invention and a second stereoscopic image depicting the characteristics of the object to be photographed. Figure 9 The second stereoscopic image shown is Figure 5 and Figure 6 the same as the object to be photographed shown, and is an image depicting the three-dimensional shape of the object to be photographed in a posture and shape different from that of the object to be photographed. However, Figure 9 the object to be photographed depicted in the second stereoscopic image shown Figure 7 compared to the object to be photographed depicted in the second stereoscopic image shown, Figure 5 has a greater change in posture and shape with respect to the object to be photographed shown. Figure 9 The points c4, c5, and n8 shown are all examples of second feature points represented by second reference data, and are points on the cross-section of the bone on the surface of the object to be photographed.

[0057] In addition, Figure 9 the points c4, c5, and n8 shown in Figure 6 are respectively made to correspond to the points c4, c5, and n8 shown in Figure 6 by using feature-based matching of three feature points. And the correspondence of these three points determines the relationship between the posture and shape of the object to be photographed depicted in the first stereoscopic image shown in Figure 9 and the posture and shape of the object to be photographed depicted in the second stereoscopic image shown in

[0058] Figure 10 This is a diagram showing an example of the part where the feature represented by the feature data is located in the object to be photographed shown in Figure 9 The image shown in Figure 10 is the same as the object to be photographed shown in Figure 9 and is an X-ray image depicting the object to be photographed in the same posture and the same shape as the object to be photographed. Figure 10 The points A, J, C, D, E, F, G, H, N8, C5, C4, N7, and C1 shown respectively correspond to Figure 5 the points A, J, C, D, E, F, G, H, N8, C5, C4, N7, and C1 shown.

[0059] Next, an example of the processing executed by the image processing apparatus according to the embodiment will be described with reference to Figure 11 This is a flowchart showing an example of the processing executed by the image processing apparatus according to the embodiment of the present invention. In addition, the image processing apparatus 100 executes Figure 11 before the processing executed by the multi-joint robot 31 and the multi-joint robot 32. Figure 11The processing shown below. Additionally, the image processing apparatus 100 may also temporarily stop the processing of the multi-joint robot 31 and the multi-joint robot 32 to execute Figure 11 the processing shown below.

[0060] In step S10, the acquisition unit 101 acquires an X-ray image, a first reference image, and a second reference image.

[0061] In step S20, the determination unit 102 extracts feature data representing the features of the object P depicted in the X-ray image.

[0062] In step S30, the determination unit 102 extracts first reference data representing the features of the object P depicted in the first reference image.

[0063] In step S40, the determination unit 102 uses the first reference data to determine the part where the feature represented by the feature data is located in the first reference image.

[0064] In step S50, the determination unit 102 extracts second reference data representing the features of the object P depicted in the second reference image.

[0065] In step S60, the determination unit 102 uses at least one of the relationship between the first pose of the object P derived from the first reference data and the second pose of the object P derived from the second reference data and the relationship between the first shape of the object P derived from the first reference data and the second shape of the object P derived from the second reference data to determine the part where the feature represented by the feature data is located in the object P taking at least one of the second pose and the second shape.

[0066] The image processing apparatus 100 of the embodiment has been described above. The image processing apparatus 100 extracts feature data from the object P depicted in the image, extracts first reference data from the object P depicted in the first reference image, and extracts second reference data from the object P depicted in the second reference image. The image processing apparatus 100 uses the first reference data to determine the part where the feature represented by the feature data is located in the first reference image. Then, the image processing apparatus 100 uses at least one of the relationship between the first pose of the object P and the second pose of the object P and the relationship between the first shape of the object P and the second shape of the object P to determine the part where the feature represented by the feature data is located in the object P taking at least one of the second pose and the second shape.

[0067] In addition, the image processing device 100 determines a first pose of the photographed object P depicted in the first reference image using first reference data, and determines a second pose of the photographed object P depicted in the second reference image. In addition, the image processing device 100 extracts first reference data representing two first feature points located on a first straight line and a first feature point different from the points located on the first straight line, and extracts second reference data representing two second feature points located on a second straight line and a second feature point different from the points located on the second straight line.

[0068] Therefore, even after at least one of the posture and shape of the photographed object changes, the image processing device 100 can use an image taken before at least one of the posture and shape of the photographed object P changes. In addition, thereby, when the image processing device 100 is applied to the food processing line 1, the degree of freedom in the layout of the food processing line 1 can be increased. That is, the image processing device 100 can reduce the number of X-ray imaging devices or devices that shield X-rays.

[0069] In addition, the image processing device 100 extracts at least one of a first feature point that remains fixed in position in the photographed object P even when a predetermined process is performed on the photographed object P and a second feature point that remains fixed in position in the photographed object P even when a predetermined process is performed on the photographed object P. Therefore, even when a predetermined process is performed on the photographed object P so that the shape, position, posture, etc. of the photographed object P change, the image processing device 100 can more accurately extract the first feature point and the second feature point, and can more accurately perform processing using the first feature point and the second feature point.

[0070] In addition, in the above-described embodiment, an example is given in which the image processing device 100 extracts feature data from an X-ray image depicting a photographed object P having a predetermined pose and a predetermined shape, but the present invention is not limited thereto. The image processing device 100 may not extract feature data from an X-ray image taken using the X-ray imaging device 20, but may extract feature data, for example, from a CT image obtained by photographing the photographed object P using an X-ray CT (Computed Tomography) device or an MRI image obtained by photographing the photographed object P using a magnetic resonance imaging (MRI) device. Alternatively, the image processing device 100 may extract feature data from an ultrasonic image, an infrared image, a microwave image, an ultraviolet image, or a terahertz image obtained by photographing the photographed object P.

[0071] In addition, even when at least one of the posture and shape of the object P changes at a position different from the position where the image is captured, a capturing device such as the X-ray capturing device 20 that captures the image for extracting the feature data is preferably provided only at one part on the food processing line 1. This is because, when multiple such capturing devices are provided, the overall management cost of the food processing line 1 increases, and the processing load for generating the image increases, resulting in a decrease in production efficiency.

[0072] In addition, when such a capturing device emits radiation, the food processing line 1 requires a device for shielding the radiation in order to prevent actuators used in robots such as the multi-joint robot 31 and the multi-joint robot 32 from performing unwanted actions due to the radiation. Therefore, the food processing line 1 needs to ensure space for installing this device, and the management cost of this device is added to the overall management cost. Moreover, if the radiation shielding device includes substances such as lead that need to be prevented from mixing into food, it is difficult to install this device on the food processing line 1.

[0073] In addition, in the above-described embodiment, as an example, a case where the position for capturing the first stereoscopic image is different from the position for capturing the second stereoscopic image is given, but this is not limitative. That is, the position for capturing the first stereoscopic image and the position for capturing the second stereoscopic image may be the same.

[0074] In addition, in the above-described embodiment, as an example, a case where the first stereoscopic image is captured by the first stereoscopic image capturing device 21 and the second stereoscopic image is captured by the second stereoscopic image capturing device 22 is given, but this is not limitative. That is, the first stereoscopic image and the second stereoscopic image may also be captured by the same capturing device.

[0075] In addition, in the above-described embodiment, as an example, a case where the feature data is data representing feature points on the object P is given, but this is not limitative. For example, the feature data may also be data representing a feature region that is a feature region located on the object P depicted in the X-ray image, or a feature shape that is a feature shape of the object P.

[0076] Moreover, in the above-described embodiment, as an example, a case where the first reference data is data representing the first feature points on the object P is given, but this is not limitative. For example, the first reference data may also be data representing a first feature region that is a feature region located on the object P depicted in the first reference image, or a first feature shape that is a feature shape of the object P.

[0077] In addition, in the above-described embodiments, as an example, the case where the second reference data is data representing the second feature point on the subject P has been described, but it is not limited thereto. For example, the second reference data may also be data representing a second feature region that is a feature region on the subject P depicted in the second reference image, or a second feature shape that is a feature shape of the subject P.

[0078] In addition, in the above-described embodiments, the case where the first reference image is the first stereoscopic image has been described as an example, but it is not limited thereto. That is, the first reference image may also be an image other than the first stereoscopic image that stereoscopically depicts the subject P.

[0079] In addition, in the above-described embodiments, the case where the second reference image is the second stereoscopic image has been described as an example, but it is not limited thereto. That is, the second reference image may also be an image other than the second stereoscopic image that stereoscopically depicts the subject P.

[0080] In addition, in the above-described embodiments, the case where feature-based matching is applied to the first stereoscopic image and the second stereoscopic image to establish correspondence between the first feature point and the second feature point has been described as an example, but it is not limited thereto. The image processing apparatus 100 may also apply region-based matching to the first stereoscopic image and the second stereoscopic image to establish correspondence between the first feature point and the second feature point. Region-based matching is a technique for searching for similar regions between multiple images and using those regions to match the multiple images.

[0081] As described above, the embodiments of the present invention have been described with reference to the accompanying drawings. However, the image processing apparatus 100 is not limited to the above-described embodiments, and various modifications, substitutions, combinations, or design changes can be made without departing from the gist of the present invention.

[0082] Reference Numerals

[0083] 100... Image processing apparatus 101... Acquisition unit 102... Determination unit.

Claims

1. An image processing apparatus, characterized in that, comprising: an acquisition unit that acquires an X-ray image, a first reference image, and a second reference image, wherein the X-ray image is an X-ray image of a subject having a predetermined posture and shape with an internal structure, taken at a first position having an X-ray shielding device, the first reference image is taken at the first position, includes a cross-section of the internal structure, and is an image of the subject having a first posture known in relation to the predetermined posture and a first shape known in relation to the predetermined shape, the second reference image is an image of the subject taken at a second position, includes a cross-section of the internal structure, and is an image of the subject having at least one of a second posture different from the predetermined posture and a second shape different from the predetermined shape, the second position being a position to which the conveyor transports the subject from the first position and being a position of a robot that cuts the meat contained in the subject and removes the tail bone and hip bone from the subject; and a determination unit that extracts feature data representing the features of the subject depicted in the X-ray image, extracts first reference data representing the features of the subject depicted in the first reference image, extracts second reference data representing the features of the subject depicted in the second reference image, and determines, using the first reference data, the part where the feature represented by the feature data is located in the first reference image, and determines, using at least one of the relationship between the first posture of the subject derived from the first reference data and the second posture of the subject derived from the second reference data and the relationship between the first shape of the subject derived from the first reference data and the second shape of the subject derived from the second reference data, the part where the feature represented by the feature data is located in the subject having at least one of the second posture and the second shape.

2. The image processing apparatus according to claim 1, characterized in that, the determination unit performs at least one of a process of determining the first posture using the first reference data and a process of determining the second posture using the second reference data.

3. The image processing apparatus according to claim 2, characterized in that, the determination unit extracts at least one of the first reference data and the second reference data, wherein the first reference data represents a feature whose position in the subject remains fixed even when a predetermined process is performed on the subject, and the second reference data represents a feature whose position in the subject remains fixed even when the predetermined process is performed on the subject.

4. The image processing apparatus according to claim 2 or 3, characterized in that, The determination unit performs at least one of a process of extracting the first reference data and a process of extracting the second reference data, where the first reference data represents two first feature points located on a first straight line and the first feature points different from the points located on the first straight line, and the second reference data represents two second feature points located on a second straight line and the second feature points different from the points located on the second straight line.

5. A storage medium, characterized in that, an image processing program is recorded on the storage medium, the image processing program causes a computer to implement an acquisition function and a determination function, where, the acquisition function is used to acquire an X-ray image, a first reference image, and a second reference image, where the X-ray image is an X-ray image of a subject having a predetermined posture and a predetermined shape with an internal structure, taken at a first position with an X-ray shielding device, the first reference image is taken at the first position, includes a cross-section of the internal structure, and the subject taken in a first posture having a known relationship with the predetermined posture and a first shape having a known relationship with the predetermined shape is photographed, the second reference image is an image depicting the subject taken at a second position, includes a cross-section of the internal structure, and the subject taken in at least one of a second posture different from the predetermined posture and a second shape different from the predetermined shape is photographed, the second position is a position to which the conveyor belt conveys the subject from the first position and is a position of a robot configured to cut the meat contained in the subject and remove the tail bone and hip bone from the subject, and the determination function extracts feature data representing the features of the subject depicted in the X-ray image, extracts first reference data representing the features of the subject depicted in the first reference image, extracts second reference data representing the features of the subject depicted in the second reference image, and determines the part where the feature represented by the feature data is located in the first reference image by using the first reference data, and determines the part where the feature represented by the feature data is located in the subject taking at least one of the second posture and the second shape by using at least one of the relationship between the first posture of the subject derived from the first reference data and the second posture of the subject derived from the second reference data, and the relationship between the first shape of the subject derived from the first reference data and the second shape of the subject derived from the second reference data.

6. An image processing method, characterized in that, includes an acquisition step and a determination step, where, In the obtaining step, an X-ray image, a first reference image, and a second reference image are obtained. The X-ray image is an X-ray image of a subject having a predetermined posture and a predetermined shape with an internal structure, taken at a first position with an X-ray shielding device. The first reference image is taken at the first position, includes a cross-section of the internal structure, and is an image of the subject having a first posture known in relation to the predetermined posture and a first shape known in relation to the predetermined shape. The second reference image is an image of the subject taken at a second position, includes a cross-section of the internal structure, and is an image of the subject having at least one of a second posture different from the predetermined posture and a second shape different from the predetermined shape. The second position is a position to which the conveyor transports the subject from the first position and is a position where a robot for cutting the meat contained in the subject and removing the tailbone and hipbone from the subject is arranged, and In the determining step, feature data representing the features of the subject depicted in the X-ray image is extracted, first reference data representing the features of the subject depicted in the first reference image is extracted, second reference data representing the features of the subject depicted in the second reference image is extracted, and the part where the feature represented by the feature data is located in the first reference image is determined by using the first reference data. The part where the feature represented by the feature data is located in the subject having at least one of the second posture and the second shape is determined by using at least one of the relationship between the first posture of the subject derived from the first reference data and the second posture of the subject derived from the second reference data, and the relationship between the first shape of the subject derived from the first reference data and the second shape of the subject derived from the second reference data.

Images