Object recognition device, object recognition system, and object recognition method

Abstract

This object recognition device comprises: a recognition unit that is disposed upstream on a path along which a target object is transported; and a processing unit that is disposed downstream of the recognition unit along the path. The recognition unit includes: a fluoroscopic image acquisition unit that acquires a fluoroscopic image in which the internal structure of the target object is captured; a first three-dimensional information acquisition unit that acquires three-dimensional information indicating the outer shape of the target object; and a recognition control unit that extracts feature points from the fluoroscopic image. The processing unit includes: a second three-dimensional information acquisition unit that acquires three-dimensional information indicating the outer shape of the target object at a position downstream on the path with respect to the position where the three-dimensional information has been acquired by the recognition unit; and a processing control unit that performs coordinate conversion of the feature points on the basis of a conversion matrix obtained as a result of matching between the three-dimensional information acquired by the first three-dimensional information acquisition unit and the three-dimensional information acquired by the second three-dimensional information acquisition unit.

Description

Object recognition device, object recognition system, and object recognition method

[0001] This application claims priority to Japanese Patent Application No. 2024-123317, filed on July 30, 2024, the contents of which are incorporated herein by reference.

[0002] Conventionally, in meat processing systems, there has been a technology that uses feature values ​​extracted from fluoroscopic images taken using an X-ray imaging device or the like to understand the internal structure of meat and perform processes such as cutting. According to this conventional technology, a device for taking fluoroscopic images, such as an X-ray imaging device, is not installed in the processing section where processing is performed, but is instead concentrated in an upstream recognition section, thereby constructing a safe and inexpensive system. In this method, since the posture of the meat to be processed changes during transportation, the recognition section and the processing section grasp the correlation between the posture of the meat to be processed in order to use the feature values ​​obtained from the fluoroscopic images obtained by the recognition section in the processing section (see, for example, Patent Document 1).

[0003] Republished WO2020 / 246288

[0004] According to a conventional method such as that described in Patent Document 1, the recognition unit and processing unit each perform three-dimensional external measurement, detect the same three feature points on the workpiece, and perform coordinate conversion. Of the three detected feature points, one is a reference point used for translational movement, and the other two are points used for rotational movement. This configuration makes it possible to calculate the amount of movement of the recognition unit's coordinate system so that it matches the processing unit's coordinate system, and to convert the feature values ​​extracted by the recognition unit into the processing unit's coordinate system.

[0005] When the recognition unit and processing unit detect the same feature point, errors may occur due to errors in three-dimensional external measurement or mistakes in image processing. In particular, when the positions of three feature points are close to each other, even a small error can result in a large error when converted into the amount of movement in a coordinate system. Another issue is that the farther the feature point is from the reference point, the larger the error when converted into the amount of movement in a coordinate system. The larger the error when converted into the amount of movement in a coordinate system, the more likely it is to have a negative impact on the success rate and yield during processing.

[0006] Furthermore, the three feature points to be detected must be able to be extracted from the external shape information and must not deform while the processing object is being transported. Therefore, when processing meat or other foods, the feature points to be detected are defined on exposed bones or at positions based on the bones. However, if there are no exposed bones on the processing object, there is an issue in that the three feature points cannot be defined.

[0007] Aspects of the present invention provide an object recognition device, an object recognition system, and an object recognition method that are capable of accurately recognizing the internal structure of an object.

[0008] [1] One aspect of the present invention is an object recognition device that includes a recognition unit that is arranged upstream of a route along which a target object is transported, and a processing unit that is arranged downstream of the recognition unit along the route. The recognition unit includes a perspective image acquisition unit that acquires a perspective image of the internal structure of the target object, a first three-dimensional information acquisition unit that acquires three-dimensional information indicating the outline of the target object, and a recognition control unit that extracts feature points from the perspective image. The processing unit includes a second three-dimensional information acquisition unit that acquires three-dimensional information indicating the outline of the target object at a point downstream of the route from the point where the three-dimensional information was acquired by the recognition unit, and a processing control unit that performs coordinate transformation of the feature points based on a transformation matrix obtained as a result of matching the three-dimensional information acquired by the first three-dimensional information acquisition unit with the three-dimensional information acquired by the second three-dimensional information acquisition unit.

[0009] According to the above-described aspect, the upstream recognition unit and the downstream processing unit each acquire three-dimensional information, and perform coordinate transformation of feature points based on the transformation matrix obtained as a result of matching the three-dimensional information. Therefore, according to this aspect, three-point detection is not performed as in conventional methods. Instead of detecting three feature points, changes in the posture of the target object are grasped by matching the entire three-dimensional information, which solves the problem of larger errors occurring for features farther from the reference point. Therefore, according to this aspect, it is possible to accurately recognize the internal structure of an object.

[0010] Furthermore, according to this aspect, since the detection of three feature points based on bone portions as in the conventional technology is not performed, it is possible to apply it to target objects in which bones are not exposed. Furthermore, according to this aspect, it is not necessary to develop an algorithm for detecting three feature points, and it is possible to realize a system with a small number of steps. Note that the conventional method may be used as the algorithm for matching the three-dimensional information itself.

[0011] [2] In one aspect of the present invention, in the object recognition device of [1] described above, the three-dimensional information acquired by the first three-dimensional information acquisition unit and the three-dimensional information acquired by the second three-dimensional information acquisition unit are both point cloud data, and the processing control unit performs matching based on the distance and relative direction of the point clouds.

[0012] According to the above-described aspect, point cloud data is used as three-dimensional information, and matching is performed based on the distance and relative direction of the point cloud. That is, according to this aspect, point cloud data representing the entire target object is used, and matching is performed based on the distance and relative direction of the point cloud. Therefore, according to this aspect, the problem of errors increasing as the feature is farther from the reference point does not occur, and it is possible to accurately recognize the internal structure of the object.

[0013] [3] In one aspect of the present invention, in the object recognition device of the above-mentioned [2], the processing control unit performs matching based on the distance and relative direction of a point cloud obtained by thinning out information from the acquired point cloud data.

[0014] That is, according to this aspect, the obtained three-dimensional information is not used as is, but the amount of information is first reduced so that it is suitable for the matching process, and then the processing is performed. Therefore, according to this aspect, the efficiency of the matching process can be improved (the amount of processing can be reduced).

[0015] [4] In one aspect of the present invention, in the object recognition device of [3] described above, the processing control unit performs a first matching based on the distance and relative direction of point cloud data, which is a point cloud obtained by thinning out information from the acquired point cloud data and has a first amount of information, and then performs a second matching based on the distance and relative direction of point cloud data, which is a point cloud obtained by thinning out information from the acquired point cloud data and has a second amount of information that is greater than the first amount of information.

[0016] That is, according to this aspect, the obtained 3D information is not used as is, but the amount of information is first reduced to make it suitable for the matching process, and then a first matching process is performed. The first matching process can also be described as sparse fine adjustment. Furthermore, according to this aspect, after the sparse fine adjustment, a second matching process is performed using point cloud data with an amount of information greater than the amount of information used in the first matching process. The second matching process can also be described as dense fine adjustment. The dense fine adjustment allows matching to be performed at a more detailed resolution. Therefore, according to this aspect, it is possible to accurately recognize the internal structure of an object while increasing the efficiency of the matching process (reducing the processing amount).

[0017] [5] In one aspect of the present invention, in the object recognition device described in any one of [1] to [4] above, in the recognition unit, a perspective image capturing device that captures the perspective image and a first three-dimensional information capturing device that captures three-dimensional information showing the outline of the target object are arranged at different positions on the path, and the recognition control unit aligns the coordinates of the perspective image and the three-dimensional information according to the distance the target object has been transported along the path.

[0018] According to the above-described aspect, if the recognition unit can capture the perspective image and the three-dimensional information at the same position on the path, it is not necessary to align the coordinates of the perspective image and the three-dimensional information. However, this is difficult in reality. Therefore, according to this aspect, the recognition unit can simply regard the perspective image and the three-dimensional information as being in the same coordinate system, even though they are actually located at different positions. By adopting such a configuration, the perspective image and the three-dimensional information can be obtained in corresponding coordinate systems through simple processing.

[0019] [6] In one aspect of the present invention, in the object recognition device according to any one of [1] to [5] above, the perspective image is an X-ray image.

[0020] According to the above-described aspect, the plurality of processing units do not require X-ray cameras for capturing X-ray images, which reduces the overall system cost and the installation area of ​​the entire system.

[0021] [7] In one aspect of the present invention, in the object recognition device described in any one of [1] to [6] above, the target object is meat, and the path is a belt conveyor that transports the meat.

[0022] According to the above-described aspect, in a system in which meat is transported on a belt conveyor and processed such as cutting, the processing unit can recognize the internal structure of the meat, such as bones, without taking a fluoroscopic image, and can perform cutting processes and the like in an appropriate manner.

[0023] [8] In one aspect of the present invention, in the object recognition device described in [7] above, the feature points are cutting points that serve as landmarks when cutting the meat.

[0024] According to the above-described aspect, in a system in which meat is transported on a belt conveyor and subjected to processes such as cutting, the meat can be suitably cut based on the obtained cutting points.

[0025] [9] One aspect of the present invention is an object recognition system including an object recognition device described in any one of [1] to [8] above, comprising one recognition unit and a plurality of processing units, wherein the path branches to each of the plurality of processing units located downstream when the recognition unit is considered to be upstream, and the target object is transported from the recognition unit to one of the processing units.

[0026] According to the above-described aspect, by providing one recognition unit and multiple processing units, it is sufficient that only the one recognition unit has a configuration for capturing a fluoroscopic image. In other words, the multiple processing units can perform processing using the fluoroscopic image captured by the one recognition unit. Therefore, according to this aspect, it is possible to reduce the cost of the entire system and the installation area of ​​the entire system.

[0027]

[10] One aspect of the present invention is the object recognition system described in [9] above, wherein the plurality of processing units perform cutting of the target object based on the feature points.

[0028] According to the above-described aspect, a system is provided with one recognition unit and multiple processing units, and cutting is performed by the multiple processing units. In other words, this aspect is a system that includes multiple processing units that perform cutting processing. By providing one recognition unit, the internal structure of the target object is recognized upstream and the internal structure is reused downstream. By adopting such a configuration, this aspect can reduce the cost of the entire system and the installation area of ​​the entire system.

[0029]

[11] One aspect of the present invention is an object recognition method comprising: a recognition step performed upstream of a route along which a target object is transported; and a processing step performed downstream of the recognition step along the route. The recognition step comprises a perspective image acquisition step of acquiring a perspective image of the internal structure of the target object, a first three-dimensional information acquisition step of acquiring three-dimensional information indicating the outline of the target object, and a recognition control step of extracting feature points from the perspective image. The processing step comprises a second three-dimensional information acquisition step of acquiring three-dimensional information indicating the outline of the target object at a point downstream of the route from the point where the three-dimensional information was acquired in the recognition step, and a processing control step of performing coordinate transformation of the feature points based on a transformation matrix obtained as a result of matching the three-dimensional information acquired by the first three-dimensional information acquisition step with the three-dimensional information acquired by the second three-dimensional information acquisition step.

[0030] According to aspects of the present invention, it is possible to provide an object recognition device, an object recognition system, and an object recognition method that are capable of accurately recognizing the internal structure of an object.

[0031] FIG. 1 is a diagram for explaining an example of an overview of an object processing system according to an embodiment. FIG. 1 is a diagram for explaining an example of an overview of a recognition unit according to the embodiment. FIG. 2 is a functional configuration diagram showing an example of the functional configuration of a recognition unit according to the embodiment. FIG. 2 is a diagram for explaining an example of an overview of a processing unit according to the embodiment. FIG. 3 is a functional configuration diagram showing an example of the functional configuration of a processing unit according to the embodiment. FIG. 3 is a diagram for explaining an example of matching according to the embodiment. FIG. 4 is a flowchart showing an example of a series of processing flows in an object processing system according to the embodiment. FIG. 4 is a diagram for explaining a modified example of the overview of an object processing system according to the embodiment. FIG. 5 is a block diagram showing the internal configuration of each device included in the object processing system according to the embodiment.

[0032] [Embodiments] Preferred embodiments of an object recognition device, an object recognition system, and an object recognition method according to aspects of the present invention will be described in detail below with reference to the accompanying drawings. Note that aspects of the present invention are not limited to these embodiments and include various modifications or improvements. In other words, the components described below include those that can be easily imagined by a person skilled in the art or that are substantially identical, and the components described below can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the present invention. Furthermore, in the drawings below, the scale and number of components may differ from the scale and number of the actual structures to make each configuration easier to understand.

[0033] [Object Processing System 1] Fig. 1 is a diagram illustrating an example of an overview of an object processing system according to an embodiment. First, an overview of the object processing system 1 will be described with reference to the diagram. The object processing system 1 includes one recognition unit 10 and at least one or more processing units 20. In the illustrated example, one processing unit 20 is provided for one recognition unit 10.

[0034] The object processing system 1 processes a target object OBJ. Specifically, the object processing system 1 performs object recognition of the target object OBJ and then processes the target object OBJ. Among the object processing system 1, a device having a configuration for performing object recognition may be referred to as an object recognition device.

[0035] The recognition unit 10 and the processing unit 20 are connected by a belt conveyor CB. The belt conveyor CB may also be referred to as a path along which a target object OBJ is transported. The transport direction of the belt conveyor CB is indicated by an arrow AR. For example, a target object OBJ is placed on the belt conveyor CB and transported along the arrow AR. The transport of the belt conveyor CB may be for transporting a placed target object OBJ or a suspended target object OBJ.

[0036] Here, the target object OBJ is an object that is the target of object recognition in this embodiment. Specifically, the target object OBJ may be meat that is to be processed. The target object OBJ preferably has an internal structure that can be grasped using a perspective image and cannot be grasped from the outside. The internal structure that can be grasped using a perspective image may be, for example, bones present inside the meat that can be imaged using X-rays. However, this embodiment is not limited to this example, and various objects can be applied as the processing target.

[0037] As shown in the figure, a recognition unit 10 is installed upstream of the belt conveyor CB. A target object OBJ transported by the belt conveyor CB always passes through the recognition unit 10. The recognition unit 10 acquires three-dimensional information and an internal structure of the target object OBJ. After passing through the recognition unit 10, the target object OBJ is transported to a processing unit 20.

[0038] The target object OBJ is always conveyed to the processing unit 20 after passing through the recognition unit 10. The processing unit 20 can also be said to be arranged downstream of the recognition unit 10.

[0039] The path from the recognition unit 10 to the processing unit 20 may not be a straight line depending on the design of the object processing system 1. In the example shown in the figure, the direction changes by 90 degrees when branching from the belt conveyor CB to the processing unit 20. When the target object OBJ is transported along such a path, the orientation grasped by the processing unit 20 may have changed. Therefore, it is necessary to grasp the orientation of the object again before processing is performed by the processing unit 20.

[0040] When the target object OBJ is transported to the processing unit 20, first, three-dimensional information of the target object OBJ is acquired. Based on the three-dimensional information acquired by the processing unit 20 and the three-dimensional information and perspective image acquired by the recognition unit 10, the internal structure in the orientation at the processing unit 20 is estimated.

[0041] Here, in order to compare information on the target object OBJ recognized by the recognition unit 10 with information on the target object OBJ recognized by the processing unit 20, an identification number may be assigned to the target object OBJ. The identification number may be assigned, for example, to a tray (not shown) on which the target object OBJ is placed, or a predetermined feature of the target object OBJ may be used as identification information using object recognition technology. Furthermore, identification may be performed according to the transport distance of the belt conveyor CB, route branch instruction information, etc.

[0042] [Recognition unit 10] Fig. 2 is a diagram illustrating an example of an overview of the recognition unit according to this embodiment. An example of an overview of the recognition unit 10 will be described with reference to the same figure. The recognition unit 10 includes an X-ray camera 11, a 3D camera 12, and a control unit 13. Note that the illustrated example is a schematic diagram for explanation purposes and does not limit the arrangement of the components.

[0043] The target object OBJ is placed on a belt conveyor CB and is transported in the direction of the arrow AR. When the target object OBJ is transported to a point P11, an X-ray image is taken by the X-ray camera 11.

[0044] The X-ray camera 11 is an example of a configuration for capturing an image of the internal structure. Specifically, the X-ray camera 11 captures an X-ray image as an example of a fluoroscopic image of the internal structure. The X-ray camera 11 can also be called a fluoroscopic image capturing device. The X-ray camera 11 outputs the captured X-ray image to the control unit 13.

[0045] After a perspective image of the target object OBJ is captured at point P11, the target object OBJ is transported to point P12. At the point when the target object OBJ is transported to point P12, three-dimensional information is acquired by the 3D camera 12.

[0046] The 3D camera 12 is an example of a configuration for acquiring the above-mentioned three-dimensional information. Specifically, the 3D camera 12 acquires a distance image having distance information at each coordinate in an x-y coordinate system as an example of three-dimensional information. The three-dimensional information acquired by the 3D camera 12 may also be point cloud data. In the following description, the 3D camera 12 may be referred to as a first three-dimensional information imaging device.

[0047] The control unit 13 acquires fluoroscopic images from the X-ray camera 11 and acquires three-dimensional information from the 3D camera 12. The control unit 13 associates the acquired information and stores the information in which the fluoroscopic images and the three-dimensional information are associated in a predetermined storage unit. An example of the functional configuration of the control unit 13 will be described below with reference to FIG. 3.

[0048] 3 is a functional configuration diagram showing an example of the functional configuration of the recognition unit according to this embodiment. Referring to the diagram, an example of the functional configuration of the control unit 13 will be described. The control unit 13 has, as its functional configuration, a perspective image acquisition unit 131, a first three-dimensional information acquisition unit 132, and a recognition control unit 133.

[0049] The fluoroscopic image acquisition unit 131 acquires a fluoroscopic image capturing an image of the internal structure of the target object OBJ. Specifically, the fluoroscopic image may be an X-ray image captured by the X-ray camera 11. Note that in this embodiment, the fluoroscopic image is not limited to an X-ray image. The fluoroscopic image may be any image that includes information about the internal structure of the object that is difficult to grasp from its external appearance.

[0050] The first three-dimensional information acquisition unit 132 acquires three-dimensional information indicating the outer shape of the target object OBJ. Specifically, the three-dimensional information may be information captured by the 3D camera 12. The three-dimensional information acquired by the first three-dimensional information acquisition unit 132 may also be referred to as first three-dimensional information.

[0051] The recognition control unit 133 first extracts feature points from the perspective image acquired by the perspective image acquisition unit 131. Feature points are points used when processing the target object OBJ. For example, if the target object OBJ is meat, the feature points may be cutting points that serve as landmarks when cutting the meat. The recognition control unit 133 associates the first three-dimensional information with the feature points and stores them in the storage unit 30. The recognition control unit 133 may further associate the first three-dimensional information with identification information of the target object OBJ for which measurement has been performed and store them in the storage unit 30.

[0052] Here, in the recognition unit 10, it is preferable to place the X-ray camera 11 and the 3D camera 12 at positions on the path where they can be considered to be the same, but this may be difficult due to installation location constraints. That is, the X-ray camera 11 and the 3D camera 12 may be placed at different positions on the path. The recognition control unit 133 may simply consider the X-ray camera 11 and the 3D camera 12 to be in the same coordinate system even when they are placed at different positions on the path. This is because, within the recognition unit 10, the path is a straight line and the orientation of the target object OBJ is likely to remain unchanged. In other words, the recognition control unit 133 may simply consider the X-ray camera 11 and the 3D camera 12 to be in the same coordinate system if the distance the target object OBJ has been transported on the path is short. Furthermore, if the distance the target object OBJ has been transported on the path is long, the recognition control unit 133 may convert the coordinates to the same coordinate system by performing a predetermined coordinate transformation. That is, the recognition control unit 133 may align the coordinates of the perspective image and the three-dimensional information depending on the distance the target object OBJ has been transported on the path.

[0053] [Processing Unit 20] Fig. 4 is a diagram illustrating an example of the outline of the processing unit according to this embodiment. An example of the outline of the processing unit 20 will be described with reference to the same figure. The processing unit 20 includes a 3D camera 21, a control unit 23, and a robot processing unit 25. Note that the illustrated example is a schematic diagram for explanation purposes and does not limit the arrangement of the components. Note that, although not illustrated, the processing unit 20 may include, in addition to the 3D camera 21, an RGB camera capable of capturing a visible image of the target object OBJ.

[0054] The target object OBJ is placed on a belt conveyor CB and is transported in the direction of the arrow AR. When the target object OBJ is transported to a point P21, three-dimensional information is captured by the 3D camera 21.

[0055] The 3D camera 21 is an example of a configuration for acquiring the above-mentioned three-dimensional information. Specifically, the 3D camera 21 acquires a distance image having distance information at each coordinate in an xy coordinate system as an example of three-dimensional information. The three-dimensional information acquired by the 3D camera 21 may also be point cloud data. In the following description, the 3D camera 21 may also be referred to as a second three-dimensional information imaging device.

[0056] The control unit 23 acquires three-dimensional information from the 3D camera 21. The control unit 23 also acquires information obtained by the recognition unit 10 (specifically, the first three-dimensional information and feature points described with reference to FIG. 3 ). Based on the acquired information, the control unit 23 corrects the positions of the feature points to match the posture of the target object OBJ at point P21. The control unit 23 outputs information about the corrected feature points to the robot processing unit 25.

[0057] After the three-dimensional information of the target object OBJ is acquired at point P21, the target object OBJ is transported to point P22. At the time when the target object OBJ is transported to point P22, processing is performed by the robot processing unit 25. At this time, the robot processing unit 25 performs processing on the target object OBJ using information on the feature points acquired from the control unit 23. For example, if the target object OBJ is meat, the processing on the target object OBJ may be a cutting process of the meat. An example of the functional configuration of the control unit 23 will be described below with reference to FIG. 5.

[0058] 5 is a functional configuration diagram showing an example of the functional configuration of a processing unit according to this embodiment. Referring to the diagram, an example of the functional configuration of the control unit 23 will be described. The control unit 23 has, as its functional configuration, a second three-dimensional information acquisition unit 231, a recognition information acquisition unit 232, and a processing control unit 233.

[0059] The second three-dimensional information acquisition unit 231 acquires three-dimensional information indicating the outer shape of the target object OBJ. Specifically, the three-dimensional information may be information captured by the 3D camera 21. Furthermore, the three-dimensional information is information acquired at point P21, which is a point downstream on the route from point P12, which is the point at which the first three-dimensional information was acquired by the processing unit 20. The three-dimensional information acquired by the second three-dimensional information acquisition unit 231 may also be referred to as second three-dimensional information.

[0060] The recognition information acquisition unit 232 acquires first three-dimensional information and information about feature points for the target object OBJ (transported to the processing unit 20) from the storage unit 30. Here, the recognition information acquisition unit 232 may acquire identification information for the target object OBJ transported to the processing unit 20 using a predetermined method, and use the acquired identification information to search for the first three-dimensional information and information about feature points corresponding to the identification information. The predetermined method for acquiring the identification information may be based on three-dimensional information acquired by the second three-dimensional information acquisition unit 231, or may be based on some member that moves along the path with the target object OBJ, such as a tray on which the target object OBJ is placed. Furthermore, the target object OBJ itself may be marked with identification information, for example. In this case, the identification information may be obtained by analyzing an RGB image captured by an RGB camera (not shown).

[0061] The processing control unit 233 matches the three-dimensional information acquired by the first three-dimensional information acquisition unit 132 with the three-dimensional information acquired by the second three-dimensional information acquisition unit 231. As a specific example of the matching process, for example, a known algorithm may be used. An example of a known algorithm is a surface matching algorithm. As a result of the matching process, a transformation matrix is ​​obtained. The transformation matrix converts the position and orientation of the first three-dimensional information into the position and orientation of the second three-dimensional information. The processing control unit 233 performs coordinate transformation of the feature points based on the transformation matrix obtained as a result of the matching. The processing control unit 233 outputs information about the transformed feature points to the robot processing unit 25.

[0062] Specifically, the processing control unit 233 includes a matching unit 234 and a coordinate conversion unit 235, and thereby matches the three-dimensional information acquired by the first three-dimensional information acquisition unit 132 with the three-dimensional information acquired by the second three-dimensional information acquisition unit 231. The matching unit 234 performs the matching process and outputs a transformation matrix as a result of the matching process. The coordinate conversion unit 235 performs coordinate conversion of the feature points acquired by the recognition information acquisition unit based on the transformation matrix.

[0063] Here, the processing control unit 233 may perform matching processing based on, for example, distance and relative direction. For example, the three-dimensional information acquired by the first three-dimensional information acquisition unit 132 and the three-dimensional information acquired by the second three-dimensional information acquisition unit 231 may both be point cloud data. In such a case, the processing control unit 233 may perform matching based on the distance and relative direction of each point cloud.

[0064] Here, point cloud data has a large capacity, and matching all points may require a huge amount of processing. Therefore, the processing control unit 233 may perform a thinning process on the three-dimensional information acquired by the first three-dimensional information acquisition unit 132 to create data with a small capacity. Similarly, the processing control unit 233 may perform a similar thinning process on the three-dimensional information acquired by the second three-dimensional information acquisition unit 231 to create data with a small capacity. Furthermore, the processing control unit 233 may perform a matching process based on the distance and relative direction of the point cloud from which the information has been thinned.

[0065] [Example of Matching] Fig. 6 is a diagram for explaining an example of matching according to this embodiment. With reference to the same figure, an example of matching performed by the above-mentioned processing control unit 233 will be explained. Note that a specific example of the matching processing can be surface matching processing.

[0066] First, the processing control unit 233 performs a thinning process on the first three-dimensional information (step S11). As a result of the thinning process, the amount of information in the first three-dimensional information is reduced to a first amount of information. Furthermore, the processing control unit 233 performs a thinning process on the second three-dimensional information to reduce the amount of information to the first amount of information (step S12). This thinning process is a process of reducing the volume by sampling at a predetermined interval based on the information in the point cloud data. The sampling intervals in steps S11 and S12 are assumed to be the same.

[0067] Next, the processing control unit 233 performs a matching process based on the information with the first information amount (step S13). In other words, the processing control unit 233 performs the first matching process based on the distance and relative direction of the point cloud data with the first information amount, which is a point cloud obtained by thinning out information from the acquired point cloud data. The first matching process can also be considered as a fine adjustment of the sparseness.

[0068] Next, the processing control unit 233 performs a thinning process on the first three-dimensional information (step S14). As a result of the thinning process, the amount of information in the first three-dimensional information is reduced to a second amount of information. Furthermore, the processing control unit 233 performs a thinning process on the second three-dimensional information to reduce the amount of information to the second amount of information (step S15). Here, the sampling intervals in steps S14 and S15 are assumed to be the same. The second amount of information is greater than the first amount of information. In other words, the sampling intervals in steps S14 and S15 can be said to be denser than the sampling intervals in steps S11 and S12.

[0069] Furthermore, the processing control unit 233 performs a matching process based on the information with the second amount of information (step S16). In other words, the processing control unit 233 can also be said to perform a second matching process based on the distance and relative direction of the point cloud data, which is a point cloud obtained by thinning out information from the acquired point cloud data and has the second amount of information. Because the second amount of information is greater than the first amount of information, the second matching process can also be said to be a fine adjustment of the density.

[0070] By performing the coarse fine adjustment before the fine fine adjustment, the two pieces of three-dimensional information are aligned to suitable positions, and the alignment process can be easily performed in the fine fine adjustment that is performed later.

[0071] 7 is a flowchart showing an example of a series of processing steps performed by the object processing system 1 according to this embodiment. An example of the processing steps performed by the object processing system 1 will be described with reference to the flowchart.

[0072] (Step S21) First, the perspective image acquisition unit 131 acquires a perspective image, and the first three-dimensional information acquisition unit 132 acquires three-dimensional information.

[0073] (Step S22) Next, the recognition control unit 133 performs a point extraction process based on the acquired perspective image and three-dimensional information. The point extraction process is a process of extracting feature points.

[0074] (Step S23) Next, the recognition control unit 133 creates a model based on the acquired three-dimensional information. Specifically, the model is obtained by performing a sampling process on the three-dimensional information, which is point cloud data, to reduce the amount of information. The created model is stored in the storage unit. Note that steps S22 and S23 may be performed in parallel.

[0075] (Step S24) Next, the recognition control unit 133 stores, in the storage unit 30, information about the feature points extracted in step S22 and the model created in step S23.

[0076] The processes of steps S21 to S24 described so far are performed in the recognition unit 10. Therefore, these steps may be referred to as recognition steps. The processes of steps S31 to S37 described below are performed in the respective processing units 20. Therefore, these steps may be referred to as processing steps.

[0077] (Step S31) When each processing unit 20 detects that the target object OBJ has been transported (i.e., step S31; YES), the process proceeds to step S32. A standby state is maintained until the target object OBJ is transported. Here, the transport of the target object OBJ may be detected, for example, by a sensor or the like provided at the entrance of the processing unit 20.

[0078] (Step S32) When the target object OBJ is transported to the processing unit 20, first, the second three-dimensional information acquisition unit 231 acquires three-dimensional information.

[0079] (Step S33) Next, the processing control unit 233 creates a model based on the acquired three-dimensional information. This model is a model of the same format as the model created in step S23 described above. The same format means the same data format and the same sampling interval.

[0080] (Step S34) Furthermore, based on the ID (identification information) of the transported target object OBJ, the process control unit 233 reads the model created in step S23 and information on the feature points extracted in step S22 from the storage unit 30. Here, information on a plurality of target objects OBJ is stored in the storage unit 30.

[0081] (Step S35) Furthermore, the processing control unit 233 performs a matching process based on the model created in step S33 and the model read out in step S34. As a result of the matching process, a transformation matrix can be obtained.

[0082] (Step S36) The process control unit 233 moves the feature points extracted in step S22 based on the obtained transformation matrix. The movement of the feature points can also be considered as coordinate transformation.

[0083] (Step S37) The robot processing unit 25 performs robot control based on the new feature points obtained as a result of the coordinate transformation (feature points corresponding to the posture of the target object OBJ in the processing unit 20). An example of robot control is a meat cutting process.

[0084] [Modification] Figure 8 is a diagram for explaining a modification of the outline of the object processing system according to the present embodiment. With reference to this figure, an object processing system 1A, which is a modification of the object processing system 1 described with reference to Figure 1, will be described. Note that in the description given with reference to this figure, components already described with reference to Figure 1 may be omitted by assigning the same reference numerals.

[0085] The object processing system 1 can also be said to include one recognition unit 10 and at least one or more processing units 20. In other words, a plurality of processing units 20 may be provided for one recognition unit 10. In the example shown in the figure, processing units 20-1 and 20-2 are shown as examples of the plurality of processing units 20. In the following description, when there is no need to distinguish between the processing units 20-1 and 20-2, they may be simply referred to as processing units 20.

[0086] As shown in the figure, a recognition unit 10 is installed upstream of the belt conveyor CB. A target object OBJ transported by the belt conveyor CB always passes through the recognition unit 10. After passing through the recognition unit 10, the target object OBJ is transported to each processing unit 20. The processing unit 20 to which the object is to be transported may be a processing unit 20 that is not currently performing processing (is empty), or the destination may be selected (assigned) in order.

[0087] The target object OBJ is always transported to the processing unit 20 after passing through the recognition unit 10. It can also be said that each processing unit 20 is arranged downstream of the recognition unit 10. When the recognition unit 10 is considered to be upstream, the path for transporting the target object OBJ can also be said to branch off to each of the multiple processing units 20 located downstream. The target object OBJ is transported along this path, and is transported from the recognition unit 10 to one of the processing units 20.

[0088] Here, a plurality of processing units 20 are provided, and the target object OBJ reaches the processing unit 20 from the recognition unit 10 by branching the path. That is, the path from the recognition unit 10 to the processing unit 20 may not be a straight line. When the target object OBJ is transported along such a path, the orientation grasped by the processing unit 20 may have changed. Therefore, it is necessary to grasp the orientation of the object again before processing is performed by the processing unit 20.

[0089] When multiple target objects OBJ are processed consecutively, information about the multiple target objects OBJ is stored in the storage unit 30. The target object OBJ whose information is stored in the storage unit 30 is transported to one of the multiple processing units 20 included in the object processing system 1. The processing control unit 233 provided in the processing unit 20 may then identify the transported target object OBJ and read out information about the identified target object OBJ from the storage unit 30.

[0090] [Internal Structure] FIG. 9 is a block diagram showing the internal configuration of each device included in the object processing system according to this embodiment. At least some of the functions of the recognition unit 10 or the processing unit 20 can be implemented using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices, etc. The input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906. Furthermore, all or part of the functional units provided in the recognition unit 10 or the processing unit 20 may be realized using hardware such as an ASIC, a PLD, or an FPGA. Furthermore, all or part of the functional units may be realized by a combination of software and hardware.

[0091] Note that all or part of the functions of each unit of each device included in the object processing system 1 in the above-described embodiment may be realized by recording a program for realizing these functions on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0092] Furthermore, "computer-readable recording media" refers to portable media such as optical magnetic disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over a network such as the Internet, and devices that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client in such cases. Furthermore, the program may be one that realizes part of the aforementioned functions, or may be one that can realize the aforementioned functions in combination with a program already stored in the computer system.

[0093] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the spirit of the present invention.

[0094] 1...Object processing system, 10...Recognition unit, 20...Processing unit, 11...X-ray camera, 12...3D camera, 13...Control unit, 131...Fluoroscopic image acquisition unit, 132...First three-dimensional information acquisition unit, 133...Recognition control unit, 30...Memory unit, 21...3D camera, 23...Control unit, 25...Robot processing unit, 231...Second three-dimensional information acquisition unit, 232...Recognition information acquisition unit, 233...Processing control unit, 234...Matching unit, 235...Coordinate conversion unit, OBJ...Target object, CB...Belt conveyor

Claims

1. An object recognition device comprising: a recognition unit arranged upstream of a route along which a target object is transported; and a processing unit arranged downstream of the recognition unit along the route, wherein the recognition unit comprises: a perspective image acquisition unit that acquires a perspective image of the internal structure of the target object; a first three-dimensional information acquisition unit that acquires three-dimensional information indicating the outer shape of the target object; and a recognition control unit that extracts feature points from the perspective image, wherein the processing unit comprises: a second three-dimensional information acquisition unit that acquires three-dimensional information indicating the outer shape of the target object at a point along the route downstream of the point where the three-dimensional information was acquired by the recognition unit; and a processing control unit that performs coordinate transformation of the feature points based on a transformation matrix obtained as a result of matching the three-dimensional information acquired by the first three-dimensional information acquisition unit with the three-dimensional information acquired by the second three-dimensional information acquisition unit.

2. The object recognition device according to claim 1, wherein the three-dimensional information acquired by the first three-dimensional information acquisition unit and the three-dimensional information acquired by the second three-dimensional information acquisition unit are both point cloud data, and the processing control unit performs matching based on the distance and relative direction of the point clouds.

3. The object recognition device according to claim 2, wherein the processing control unit performs matching based on the distance and relative direction of a point cloud obtained by thinning out information from the acquired point cloud data.

4. The object recognition device according to claim 3, wherein the processing control unit performs a first matching based on the distance and relative direction of point cloud data obtained by thinning out information from the acquired point cloud data and having a first amount of information, and then performs a second matching based on the distance and relative direction of point cloud data obtained by thinning out information from the acquired point cloud data and having a second amount of information greater than the first amount of information.

5. The object recognition device according to claim 1, wherein in the recognition unit, a perspective image capturing device that captures the perspective image and a first three-dimensional information capturing device that captures three-dimensional information showing the outline of the target object are arranged at different positions on the route, and the recognition control unit aligns the coordinates of the perspective image and the three-dimensional information according to the distance the target object has been transported on the route.

6. The object recognition device according to claim 1, wherein the fluoroscopic image is an X-ray image.

7. The object recognition device according to claim 1, wherein the target object is meat, and the path is a conveyor belt that transports the meat.

8. The object recognition device according to claim 7, wherein the feature points are cutting points that serve as landmarks when cutting the meat.

9. An object recognition system comprising an object recognition device according to any one of claims 1 to 8, comprising one recognition unit and a plurality of processing units, wherein the path branches off to each of the plurality of processing units located downstream when the recognition unit is considered to be upstream, and the target object is transported from the recognition unit to any one of the processing units.

10. The object recognition system according to claim 9, wherein the plurality of processing units perform cutting of the target object based on the feature points.

11. An object recognition method comprising: a recognition step carried out upstream of a route along which a target object is transported; and a processing step carried out downstream of the recognition step on the route, wherein the recognition step comprises: a perspective image acquisition step of acquiring a perspective image of the internal structure of the target object; a first three-dimensional information acquisition step of acquiring three-dimensional information indicating the outer shape of the target object; and a recognition control step of extracting feature points from the perspective image, wherein the processing step comprises: a second three-dimensional information acquisition step of acquiring three-dimensional information indicating the outer shape of the target object at a point on the route downstream of the point where the three-dimensional information was acquired in the recognition step; and a processing control step of performing coordinate transformation of the feature points based on a transformation matrix obtained as a result of matching the three-dimensional information acquired in the first three-dimensional information acquisition step with the three-dimensional information acquired in the second three-dimensional information acquisition step.

Images