Item identification device, item identification system, and program

Abstract

To enable accurate identification of the size of an article.SOLUTION: An image acquisition unit 101 acquires a color image and depth image obtained by photographing multiple markers, an article placed between the multiple markers, and an arrangement surface having the multiple markers and the article arranged thereon from a first position. A type identification unit 102 identifies the type of the article from the color image. A size identification unit 107 identifies the size of the article from the depth image.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to an item identification device, an item identification system, and a program. [Background technology]

[0002] Currently, orthopedic surgery using implants, which are artificial joints implanted in the human body, and instruments for handling the implants are known. In such orthopedic surgery, an operating room nurse, known as an instrument delivery nurse, prepares the implants and instruments according to the patient's surgical procedure and hands these implants or instruments over to the orthopedic surgeon.

[0003] Because there are a wide variety of implants and instruments, even experienced nurses have difficulty selecting the appropriate implant or instrument. Currently, various technologies are known to support such orthopedic surgery. For example, Patent Document 1 discloses a technology that uses local features to identify medical objects in captured video and detect errors, defects, etc. in the medical objects. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6226187 Summary of the Invention [Problem to be solved by the invention]

[0005] Currently known methods for detecting an object from a captured image, including the technology described in Patent Document 1, are often techniques for detecting an object from a captured color image. However, when detecting an object from a color image, even if the type of the object can be identified with relatively high accuracy, it is often difficult to identify the size of the object with high accuracy. For this reason, a technology for identifying the size of an object with high accuracy is desired.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide an item identification device and the like that can accurately identify the size of an item. [Means for solving the problem]

[0007] In order to achieve the above object, an item identification device according to a first aspect of the present disclosure includes: an image acquisition unit that acquires a first color image and a first depth image by capturing images of a plurality of markers, an object placed between the plurality of markers, and a surface on which the plurality of markers and the object are placed from a first point; a type identification unit that identifies the type of the article from the first color image; and a size determination unit that determines the size of the item from the first depth image.

[0008] an image correcting unit that corrects the first depth image into a first corrected depth image that represents a depth when the placement surface is viewed from a direction perpendicular to the placement surface; an image conversion unit that converts the first corrected depth image into a first height image that represents a height from the placement surface; an image generating unit that generates a first intra-marker image that is an image of an area inside the plurality of markers in the first height image, The size identification unit may input the image within the first marker to a trained model and identify the size of the item based on output data of the trained model.

[0009] the image acquisition unit acquires a second color image and a second depth image obtained by capturing an image of the plurality of markers, the item, and the placement surface from a second point; the image correcting unit corrects the second depth image into a second corrected depth image that represents a depth when the placement surface is viewed from a direction perpendicular to the placement surface; the image conversion unit converts the second corrected depth image into a second height image representing a height from the placement surface; the image generation unit generates a second intra-marker image, which is an image of an area inside the plurality of markers in the second height image; The size identification unit may input the second intra-marker image to the trained model and identify the size of the item based on output data of the trained model for each of the first intra-marker image and the second intra-marker image.

[0010] an image integration unit that generates an integrated corrected depth image by integrating the first corrected depth image and the second corrected depth image; the image conversion unit converts the integrated corrected depth image into an integrated height image representing a height from the placement surface; the image generation unit generates an integrated intra-marker image, which is an image of an area inside the plurality of markers in the integrated height image; The size identification unit may input the integrated marker image into the trained model, and identify the size of the item based on output data of the trained model for each of the first marker image, the second marker image, and the integrated marker image.

[0011] The item is identified by type, size, and a side that designates one of a pair of mirror-image items; The image processing device may further include a side identifying unit that identifies a side of the article from the first depth image.

[0012] an image correcting unit that corrects the first depth image into a first corrected depth image that represents a depth when the placement surface is viewed from a direction perpendicular to the placement surface; an image conversion unit that converts the first corrected depth image into a first height image that represents a height from the placement surface; an image generating unit that generates a first intra-marker image, which is an image of an area inside the plurality of markers in the first height image; a binarization unit that generates a first binarized image by binarizing the first intra-marker image, The side identification unit may identify the side of the article based on a distribution of binarized data in a first intra-rectangle image, which is an image inside a minimum bounding rectangle that surrounds the article in the first binarized image.

[0013] the image acquisition unit acquires a second color image and a second depth image obtained by capturing an image of the plurality of markers, the item, and the placement surface from a second point; the image correcting unit corrects the second depth image into a second corrected depth image that represents a depth when the placement surface is viewed from a direction perpendicular to the placement surface; the image conversion unit converts the second corrected depth image into a second height image representing a height from the placement surface; the image generation unit generates a second intra-marker image, which is an image of an area inside the plurality of markers in the second height image; the binarization unit generates a second binarized image by binarizing the second intra-marker image; The side identification unit may identify the side of the item based on the distribution of binary data in the first rectangular image and the distribution of binary data in a second rectangular image, which is an image inside a minimum bounding rectangle that surrounds the item in the second binary image.

[0014] an image integration unit that generates an integrated corrected depth image by integrating the first corrected depth image and the second corrected depth image; the image conversion unit converts the integrated corrected depth image into an integrated height image representing a height from the placement surface; the image generation unit generates an integrated intra-marker image, which is an image of an area inside the plurality of markers in the integrated height image; the binarization unit generates an integrated binarized image by binarizing the image within the integrated marker; The side identification unit may identify the side of the item based on the distribution of binary data in the first rectangular image, the distribution of binary data in the second rectangular image, and the distribution of binary data in an integrated rectangular image, which is an image inside a minimum bounding rectangle that surrounds the item in the integrated binary image.

[0015] In order to achieve the above object, an item identification system according to a second aspect of the present disclosure includes: an imaging unit that captures images of a plurality of markers, an object placed between the plurality of markers, and a surface on which the plurality of markers and the object are placed from a first point, and generates a first color image and a first depth image; a type identification unit that identifies the type of the article from the first color image; a size identification unit that identifies a size of the item from the first depth image; The system further includes a display control unit that causes a display unit to display item information based on at least one of the type of item identified by the type identification unit and the size of the item identified by the size identification unit.

[0016] In order to achieve the above object, a program according to a third aspect of the present disclosure includes: Computer, an image acquisition unit that acquires a first color image and a first depth image by capturing images of a plurality of markers, an object placed between the plurality of markers, and a surface on which the plurality of markers and the object are placed from a first point; a type identification unit that identifies the type of the article from the first color image; The object detection unit functions as a size determination unit that determines the size of the object from the first depth image. [Effects of the Invention]

[0017] According to the present disclosure, the size of an article can be determined with high accuracy. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a block diagram of an item identification system according to an embodiment of the present invention; [Figure 2] 1 is a diagram illustrating a configuration of an item identification device according to an embodiment of the present invention; [Figure 3] Layout diagram of an imaging device according to an embodiment [Figure 4] Functional configuration diagram of an item identification device according to an embodiment [Figure 5] Layout diagram [Figure 6] Diagram showing depth images [Figure 7] Diagram showing the corrected depth image [Figure 8] Illustration of point cloud data correction [Figure 9] Diagram showing height image [Figure 10] Image inside the marker [Figure 11] A diagram showing a binarized image [Figure 12] Diagram showing the image within the rectangle [Figure 13] An illustration of how to identify the side of an item from the distribution of binary data. [Figure 14] 1 is a flowchart showing an item identification process executed by an item identification device according to an embodiment; [Figure 15] A flowchart showing the size specification process shown in FIG. 14. [Figure 16] A flowchart showing the side identification process shown in FIG. 14. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals.

[0020] (Embodiment) First, with reference to FIG. 1, the configuration of an item identification system 1000 according to an embodiment will be described. The item identification system 1000 is a system that identifies an item included in a captured image. The type of item to be used as the item to be identified can be adjusted as appropriate. In this embodiment, the item to be identified is an implant or an instrument used in orthopedic surgery. An implant is an artificial joint implanted in the human body. An instrument is an object used to handle an implant, such as an object that is temporarily implanted in the human body in place of an implant, or a tool used when implanting an implant in the human body. The item identification system 1000 can, for example, assist an instrument delivery nurse in appropriately selecting implants and instruments.

[0021] In this embodiment, identifying an article means identifying the type, size, and side of the article. The type of article is, for example, a concept for identifying the use of the article and does not include the size and side of the article. Examples of article types include femoral components, femoral cutting guides, femoral component trials, tibial plates, and tibial cutting guides. In this embodiment, articles of the same type basically have the same shape and are similar.

[0022] The size of an item is a concept for identifying the size of an item. The method for defining the size of an item can be adjusted as appropriate. For example, the size of an item can be expressed by the volume, height, or width of the item. For example, the size of an item can be expressed by the length of the longitudinal direction of the item. Furthermore, for example, the size of an item can be expressed by the length of the long or short side of the minimum bounding rectangle that surrounds the item when the item is placed in a manner appropriate for the item and viewed in plan. The minimum bounding rectangle, also known as the minimum bounding box, is the rectangle that has the smallest area among the rectangles that surround a certain shape.

[0023] The side of an article is a concept that designates one of a pair of articles that are mirror images of each other. For example, even if the same type of implant is implanted into the human body, there are implants for the right side of the human body and implants for the left side of the human body. Specifically, for example, there are femoral components for the right foot and femoral components for the left foot. Similarly, there are instruments for the right side and instruments for the left side that correspond to the implants.

[0024] For example, femoral component trials, which are instruments corresponding to femoral components, exist for right and left feet. The femoral component trials are instruments that are temporarily implanted in the human body before the femoral component is implanted in the body, and are used to check the size, operation, etc. of the femoral component. The shape and size of the femoral component trials are the same as those of the femoral component. In this embodiment, the side of the article is a concept that specifies either the right or left side. Note that the side of the article is not limited to the concept of specifying either the right or left side, and may also be a concept that specifies either the upper side or the lower side, or either the front side or the back side.

[0025] As described above, in this embodiment, an item is identified by the type, size, and side of the item. In other words, when comparing two items, even if the type of the item is the same, if the size or side of the item is different, the two items are different items.

[0026] 1, the item identification system 1000 includes an item identification device 100, an imaging device 200A, an imaging device 200B, and a display device 300. Hereinafter, the imaging device 200A and the imaging device 200B will be collectively referred to as the imaging device 200 as appropriate. The item identification device 100 is connected to each of the imaging device 200A, the imaging device 200B, and the display device 300 so as to be able to communicate with each other wirelessly or via a wired connection.

[0027] The item identification device 100 identifies the type, size, and side of an item captured in a color image and a depth image acquired from the imaging device 200 based on these images. A color image is an image that visualizes the color of each part in an imaging area. A color image may be a multicolor image or a monochromatic image. A multicolor image is an image that expresses the appearance of each part in an imaging area using multiple colors. A multicolor image is, for example, an RGB image that expresses the color of each part in an imaging area using the luminance values ​​of the three primary colors of RGB (Red, Green, Blue). A monochromatic image is an image that expresses the appearance of each part in an imaging area using a single color. A depth image is an image that visualizes the depth from the imaging device 200 to each part in the imaging area. For example, a depth image is an image that expresses the distance from the imaging device 200 to each part in the imaging area using the luminance value of a single color, the luminance values ​​of the three primary colors of RGB, etc.

[0028] The item identification device 100 acquires a first color image and a first depth image from the imaging device 200A, and acquires a second color image and a second depth image from the imaging device 200B. The first color image is a color image generated by the imaging device 200A. The second color image is a color image generated by the imaging device 200B. The first depth image is a depth image generated by the imaging device 200A. The second depth image is a depth image generated by the imaging device 200B. Hereinafter, the first color image and the second color image will be collectively referred to as color images, and the first depth image and the second depth image will be collectively referred to as depth images, as appropriate.

[0029] Furthermore, the item identification device 100 transmits item information, which is information related to an item, to the display device 300. The item information is information based on at least one of the item type, item size, and item side. The item information may be information indicating the item type, item size, item side, etc., or may be information derived from the item type, item size, item side, etc. For example, when two items are present in the imaging area of ​​the imaging device 200, the item information may be information indicating a determination result as to whether the two items are compatible. Furthermore, for example, when an item to be used is predetermined, the item information may be information indicating a determination result as to whether the item in the imaging area is the item to be used. The item identification device 100 is, for example, a personal computer.

[0030] As shown in FIG. 2, the item identification device 100 includes a control unit 11, a storage unit 12, a display unit 13, an operation reception unit 14, a first communication unit 15, and a second communication unit 16.

[0031] The control unit 11 includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), RTC (Real Time Clock), etc. The CPU is also called a central processing unit, central arithmetic unit, processor, microprocessor, microcomputer, DSP (Digital Signal Processor), etc., and functions as a central processing unit that executes processing and calculations related to the control of the item identification device 100. In the control unit 11, the CPU reads programs and data stored in the ROM and uses the RAM as a work area to perform overall control of the item identification device 100. The RTC is, for example, an integrated circuit with a timekeeping function. The CPU can determine the current date and time from the time information read from the RTC.

[0032] The storage unit 12 includes a nonvolatile semiconductor memory such as a flash memory, an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically Erasable Programmable ROM), and serves as a so-called auxiliary storage device. The storage unit 12 stores programs and data used by the control unit 11 to execute various processes. The storage unit 12 also stores data generated or acquired by the control unit 11 executing various processes. For example, the storage unit 12 stores a color image, a depth image, a corrected depth image, a height image, an image within a mark, a binarized image, an image within a rectangle, a trained model, etc.

[0033] The display unit 13 displays various images under the control of the control unit 11. For example, the display unit 13 displays a screen for accepting various operations from the user. The display unit 13 includes a touch screen, a liquid crystal display, etc. The operation accepting unit 14 accepts various operations from the user and supplies information indicating the contents of the accepted operations to the control unit 11. The operation accepting unit 14 includes a touch screen, a button, a lever, etc.

[0034] The first communication unit 15 communicates with the imaging device 200 under the control of the control unit 11. For example, the first communication unit 15 acquires a captured image from the imaging device 200 under the control of the control unit 11. Note that a captured image is a general term for a color image and a depth image. The first communication unit 15 communicates with the imaging device 200 in accordance with a well-known wired communication standard or a well-known wireless communication standard. Well-known wired communication standards include USB (Universal Serial Bus, registered trademark) and Thunderbolt (registered trademark). Well-known wireless communication standards include Wi-Fi (registered trademark), Bluetooth (registered trademark), Zigbee (registered trademark), etc. The first communication unit 15 includes a communication interface compliant with various communication standards.

[0035] The second communication unit 16 communicates with the display device 300 under the control of the control unit 11. For example, the second communication unit 16 transmits product information to the display device 300. The second communication unit 16 communicates with the display device 300 in accordance with a well-known wired communication standard or a well-known wireless communication standard. The second communication unit 16 includes a communication interface that complies with various communication standards.

[0036] The imaging device 200A captures an image of the imaging area to generate a first color image and a first depth image. The imaging device 200A may generate the first color image and the first depth image in response to a trigger provided by the item identification device 100, or may generate the first color image and the first depth image at a predetermined cycle. The imaging device 200A includes an image sensor for generating the color image and a depth sensor for generating the depth image. The imaging device 200A also includes a communication interface compliant with various communication standards, and communicates with the item identification device 100 in accordance with a known wired communication standard or a known wireless communication standard.

[0037] 3, the imaging device 200A is supported by a support member 251 and captures an image of an imaging area including four markers 400, an object 500 arranged between the four markers 400, and an arrangement surface 600 on which the four markers 400 and the object 500 are arranged, from a first point. The imaging area of ​​the image sensor provided in the imaging device 200A and the imaging area of ​​the depth sensor provided in the imaging device 200A almost overlap. The first color image and the first depth image are obtained by capturing images of the four markers 400, the object 500, and the arrangement surface 600.

[0038] The imaging device 200B generates a second color image and a second depth image by capturing an image of the imaging area. The configuration of the imaging device 200B is basically the same as that of the imaging device 200A. The imaging device 200B may generate the second color image and the second depth image in accordance with a trigger provided by the item identification device 100, or may generate the second color image and the second depth image at a predetermined cycle. The imaging device 200B includes an image sensor for generating a color image and a depth sensor for generating a depth image. The imaging device 200B also includes a communication interface compliant with various communication standards and communicates with the item identification device 100 in accordance with a known wired communication standard or a known wireless communication standard.

[0039] 3, the imaging device 200B is supported by a support member 252 and captures an image of an imaging area including the four markers 400, the object 500 arranged between the four markers 400, and the placement surface 600 on which the four markers 400 and the object 500 are arranged, from a second point. The imaging area of ​​the image sensor provided in the imaging device 200B and the imaging area of ​​the depth sensor provided in the imaging device 200B almost overlap. The second color image and the second depth image are obtained by capturing images of the four markers 400, the object 500, and the placement surface 600.

[0040] The four markers 400 are marks used to identify the size of the article 500 shown in the captured image. The four markers 400 are also marks used to align the captured image captured by the imaging device 200A with the captured image captured by the imaging device 200B. Marker 400 is a collective term for marker 400A, marker 400B, marker 400C, and marker 400D. Marker 400A, marker 400B, marker 400C, and marker 400D are each placed at a position corresponding to each vertex of a square having a predetermined size, for example.

[0041] The size of the article 500 in the captured image varies depending on the angle of view of the imaging device 200, the installation position of the imaging device 200, etc. However, if the size of a square with four markers 400 as vertices is constant, the size of the article 500 in real space can be calculated from the size of this square in the captured image or the length of its sides. It is preferable that the four markers 400 differ in at least one of the pattern, color, size, etc. In this embodiment, the four markers 400 differ from one another in pattern and color but are the same size. The imaging device 200 is an example of an imaging unit.

[0042] The article 500 is an article to be identified. The article 500 is placed between the four markers 400. That is, the article 500 is placed inside a square with the four markers 400 as vertices. In this embodiment, the article 500 is a femoral component trial. The placement surface 600 is a surface on which the four markers 400 and the article 500 are placed. The placement surface 600 is a floor surface, the top surface of a desk top, or the like. The placement surface 600 is basically a horizontal surface. In this embodiment, the Z axis is an axis extending vertically, the X axis is an axis perpendicular to the Z axis, and the Y axis is an axis perpendicular to the X axis and the Z axis.

[0043] The display device 300 displays various types of information. For example, the display device 300 displays information based on the item information received from the item identification device 100. The display device 300 is equipped with a touch screen, a liquid crystal display, etc., and a communication interface that complies with various communication standards. The display device 300 communicates with the item identification device 100 in accordance with a well-known wired communication standard or a well-known wireless communication standard.

[0044] Next, the functions of the item identification device 100 will be described with reference to FIG. 4. Functionally, the item identification device 100 includes an image acquisition unit 101, a type identification unit 102, an image correction unit 103, an image integration unit 104, an image conversion unit 105, an image generation unit 106, a size identification unit 107, a binarization unit 108, a side identification unit 109, a display control unit 110, and a communication control unit 111. Each of these functions is realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the ROM or storage unit 12. The CPU then executes the programs stored in the ROM or storage unit 12 to realize each of these functions.

[0045] The image acquisition unit 101 acquires a first color image and a first depth image obtained by capturing an image of the marker 400, the article 500, and the placement surface 600 from a first location. The image acquisition unit 101 also acquires a second color image and a second depth image obtained by capturing an image of the marker 400, the article 500, and the placement surface 600 from a second location from the imaging device 200B. The image acquisition unit 101 basically acquires the first color image and the first depth image from the imaging device 200A, and acquires the second color image and the second depth image from the imaging device 200B. The image acquisition unit 101 is an example of an image acquisition unit.

[0046] The type identification unit 102 identifies the type of the article 500 from the first-color image. The method by which the type identification unit 102 identifies the type of the article 500 from the first-color image can be adjusted as appropriate. For example, the type identification unit 102 may identify the type of the article 500 from the first-color image using a well-known pattern matching technique. Alternatively, the type identification unit 102 may identify the type of the article 500 from the first-color image using a well-known trained model. Note that the type identification unit 102 may identify the type of the article 500 from the second-color image, or may identify the type of the article 500 from the first-color image and the second-color image. The type identification unit 102 is an example of a type identification unit.

[0047] The image correcting unit 103 corrects the first depth image to a first corrected depth image that represents the depth when the placement surface 600 is viewed from a direction perpendicular to the placement surface 600. The image correcting unit 103 also corrects the second depth image to a second corrected depth image that represents the depth when the placement surface 600 is viewed from a direction perpendicular to the placement surface 600. The depth image represents the distance from the imaging device 200 to each part of the imaging area. Therefore, in the depth image, the depth of a portion of the placement surface 600 that is farther from the imaging device 200 is deeper than the depth of a portion of the placement surface 600 that is closer to the imaging device 200. However, a corrected depth image that is corrected to represent the entire placement surface 600 at the same depth may be easier to use than a depth image that represents different depths depending on the location on the placement surface 600. Therefore, the image correcting unit 103 corrects the depth image to a corrected depth image. The corrected depth image is a general term for the first corrected depth image and the second corrected depth image.

[0048] The first corrected depth image, which is a corrected depth image, will be described below with reference to Figures 5, 6, and 7. Figure 5 is a diagram showing the image capturing device 200, marker 400, article 500, placement surface 600, etc., viewed from the negative direction of the Y axis. As shown in Figure 5, the first point where image capturing device 200A is installed is different from the second point where image capturing device 200B is installed. In Figure 5, the first point is indicated by point 211, and the second point is indicated by point 212.

[0049] 6 is a diagram showing image 710, which is a first depth image. Image 710 is an image in which the closer the distance from image capture device 200A, that is, the shallower the depth from image capture device 200A, the darker the color. In image 710, the depth of the portion corresponding to article 500 is shallower than the depth of the portion corresponding to placement surface 600 around article 500. Also, in image 710, the depth of the portion corresponding to placement surface 600 is not constant, and the depth of the portion corresponding to placement surface 600 is shallower the closer the portion is to image capture device 200A.

[0050] 7 is a diagram showing an image 720, which is a first corrected depth image. In image 720, the depth of the portion corresponding to the article 500 is shallower than the depth of the portion corresponding to the placement surface 600 around the article 500. Also, in image 720, the depth of the portion corresponding to the placement surface 600 is constant.

[0051] The method by which the image correcting unit 103 corrects the first depth image into the first corrected depth image can be adjusted as appropriate. For example, the image correcting unit 103 identifies a portion corresponding to the placement surface 600 represented in the first depth image, and corrects the first depth image based on the depth gradient of the identified portion. Note that the image correcting unit 103 can identify the portion corresponding to the placement surface 600 represented in the first depth image using, for example, RANSAC (Random Sample Consensus). RANSAC is a method for learning parameters of a mathematical model from data including outliers while excluding the influence of outliers. In other words, the image correcting unit 103 corrects the first depth image into the first corrected depth image so that the depth gradient of the portion corresponding to the placement surface 600 represented in the first corrected depth image is eliminated.

[0052] Hereinafter, a method for correcting a depth image into a corrected depth image by correcting point cloud data will be described with reference to FIG.

[0053] In this embodiment, the depth image and the color image are represented by point cloud data. The image acquisition unit 101 may acquire point cloud data representing the depth image and the color image from the image capture device 200, or may generate the point cloud data from the depth image and the color image acquired from the image capture device 200. The point cloud data is a set of points handled by a computer.

[0054] In this embodiment, point cloud data is data that indicates the position of a point in a three-dimensional space and the color of that point. The position of a point is expressed, for example, by XYZ Cartesian coordinates. Furthermore, the color of a point is expressed, for example, by the luminance values ​​of the three primary colors, RGB. That is, in this embodiment, point cloud data is data that indicates, for each point in a three-dimensional space, the X coordinate, the Y coordinate, the Z coordinate, the red luminance value, the green luminance value, and the blue luminance value. In this way, point cloud data includes data that indicates the position and the color, but in the following explanation, the explanation of the color data will be omitted as appropriate.

[0055] 8, point 71 indicates the origin, plane 72 indicates the placement plane 600, line 73 indicates a perpendicular line extending from point 71 to plane 72, and arrow 74 indicates a normal vector of plane 72. Here, point 75 indicates p, which is the ith point in the tth frame of the point cloud data. it Also, point 76 is a point pf it Here, if the normal vector is n, an arbitrary point on the plane 72 in three-dimensional space is x, and the signed distance between the plane 72 and the point 71 is h, the plane 72 can be expressed by the formula (1) using the dot product of the vectors. n·x=h (1)

[0056] Furthermore, equation (2) is a calculation formula for projecting an arbitrary point in three-dimensional space onto the plane 72 along the normal vector, and p it From pf it Here, equation (3) is an equation showing the movement vector for projection. Here, the arithmetic mean of the movement vectors over time is calculated, and the calculated movement vector is used as a correction parameter. The point cloud data representing the depth image is corrected using this correction parameter to obtain the corrected point cloud data. The image represented by this corrected point cloud data is the corrected depth image. pf it =p it -((p it n)-h)n (2) c it =((p it n)-h)n (3)

[0057] That is, the image correcting unit 103 corrects the point cloud data representing the first depth image using the above-mentioned movement vector. The image correcting unit 103 acquires an image represented by the corrected point cloud data as a first corrected depth image. The image correcting unit 103 corrects the second depth image into a second corrected depth image using a similar method. In this way, the image correcting unit 103 corrects the depth image into a corrected depth image by correcting the point cloud data. The correction of the point cloud data is a correction for reducing a deviation caused by detection accuracy in a portion of the point cloud data corresponding to the depth image that corresponds to the placement surface 600, and is a correction for making the portion corresponding to the placement surface 600 flat according to the prerequisite that the placement surface 600 is flat. The image correcting unit 103 is an example of an image correcting unit.

[0058] The image integration unit 104 generates an integrated corrected depth image by integrating the first corrected depth image and the second corrected depth image. The method by which the image integration unit 104 generates the integrated corrected depth image can be adjusted as appropriate. For example, the image integration unit 104 applies a rigid transformation matrix to the point cloud data representing the second corrected depth image to convert the point cloud data representing the second corrected depth image into a form that can be integrated into the point cloud data representing the first corrected depth image. The rigid transformation matrix is ​​calculated based on the positions of the four markers 400 in the first corrected depth image and the positions of the four markers 400 in the second corrected depth image.

[0059] The image integration unit 104 integrates the point cloud data representing the second corrected depth image transformed by the rigid transformation with the point cloud data representing the first corrected depth image to generate point cloud data representing an integrated corrected depth image. A method for integrating depth images using point cloud data will be described below.

[0060] First, we create a point cloud data representing the first corrected depth image using pcd cam1 Then, pcd cam1 is expressed by Equation (4). Also, the point cloud data representing the second corrected depth image is expressed as pcdcam2 Then, pcd cam2 is shown by equation (5). a1 to a n represents each point whose depth is represented by the first corrected depth image. n represents each point whose depth is represented by the second corrected depth image, where n is the number of points. PCD cam1 ={a1,a2,...,a n} (4) PCD cam2 ={b1,b2,...,b n} (5)

[0061] The point cloud data corresponding to the marker 400 in the first corrected depth image is checked. cam1 Then, checker cam1 is expressed by Equation (6). The point cloud data corresponding to the marker 400 in the second corrected depth image is cam2 Then, checker cam2 is shown by equation (7). 11 , c 12 , c 13 , c 14 are points indicating the positions of the markers 400A, 400B, 400C, and 400D in the first corrected depth image, respectively. 21 , c 22 , c 23 , c 24 are points indicating the positions of the marker 400A, the marker 400B, the marker 400C, and the marker 400D, respectively, in the first corrected depth image. checker cam1 ={c 11 ,c 12 ,c 13 ,c 14} (6) checker cam2 ={c 21 ,c 22 ,c 23 ,c 24} (7)

[0062] Equation (8) is expressed as follows: R is the rotation matrix, t is the translation vector, and c 1i and c 2i where i is an integer between 1 and 4. Here, the image integration unit 104 integrates the markers 400 in the second corrected depth image so that they overlap with the markers 400 in the first corrected depth image, that is, {c 21 ,c 22 ,c 23 ,c 24} is {c 11 ,c 12 ,c 13 ,c 14}, the rotation matrix R and the translation vector t are calculated. c 1i =Rc 2i +t (8)

[0063] pcd, the composite point cloud data cam1+cam2 is expressed by equation (9). cam1+cam2 is PCD cam2 The rotation and translation operations for alignment are added to the cam1 The integrated corrected depth image is generated by adding cam1+cam2 The image is represented by the point cloud data shown below. The image integration unit 104 is an example of an image integration unit. PCD cam1+cam2 ={a1,Rb1+t,a2,Rb2+t,···,a n ,Rb n +t} (9)

[0064] The image conversion unit 105 converts the first corrected depth image into a first height image representing the height from the placement surface 600. The image conversion unit 105 also converts the second corrected depth image into a second height image representing the height from the placement surface 600. The image conversion unit 105 also converts the integrated corrected depth image into an integrated height image representing the height from the placement surface 600. The height image is an image representing the height from the placement surface 600. The height image is a collective term for the first height image, the second height image, and the integrated height image.

[0065] For example, the image conversion unit 105 performs a coordinate conversion process on the point cloud data representing the first corrected depth image to obtain point cloud data after the coordinate conversion, and obtains an image represented by the obtained point cloud data after the coordinate conversion as a first height image. This coordinate conversion is a coordinate conversion for converting a coordinate system based on the image capture device 200A into a coordinate system based on the placement surface 600. This coordinate conversion is realized by rotating the coordinate axes based on the image capture device 200A so that they overlap with the coordinate axes based on the placement surface 600. This coordinate conversion converts the point cloud data representing the distance from the image capture device 200A into point cloud data representing the height from the placement surface 600. The converted point cloud data represents the first height image. FIG. 9 shows an image 730, which is the first height image. In the image 730, the higher the height from the placement surface 600, the darker the color.

[0066] The image conversion unit 105 converts the second corrected depth image into a second height image and converts the integrated corrected depth image into an integrated height image using a similar method. For example, the image conversion unit 105 performs coordinate conversion processing on point cloud data representing the second corrected depth image to convert a coordinate system based on the image capture device 200B into a coordinate system based on the placement surface 600. The image conversion unit 105 acquires an image indicated by the point cloud data obtained by the coordinate conversion as a second height image. Furthermore, the image conversion unit 105 performs coordinate conversion processing on point cloud data representing the integrated corrected depth image to convert a coordinate system based on the image capture device 200A into a coordinate system based on the placement surface 600. The image conversion unit 105 acquires an image indicated by the point cloud data obtained by the coordinate conversion as an integrated height image. The image conversion unit 105 is an example of an image conversion unit.

[0067] The image generation unit 106 generates a first intra-marker image, which is an image of an area inside the plurality of markers 400 in the first height image. The image generation unit 106 also generates a second intra-marker image, which is an image of an area inside the plurality of markers 400 in the second height image. The image generation unit 106 also generates an integrated intra-marker image, which is an image of an area inside the plurality of markers 400 in the integrated height image. The intra-marker image is an image of an area inside four markers 400. Specifically, the intra-marker image is an image that represents the height from the placement surface 600 of the area inside the four markers 400 at a predetermined resolution. In this embodiment, the intra-marker image is an image of 150 pixels x 150 pixels, and represents the height from the placement surface 600 of a portion corresponding to each pixel by a pixel value. The intra-marker image is a collective term for the first intra-marker image, the second intra-marker image, and the integrated intra-marker image.

[0068] FIG. 10 shows image 740, which is the first intra-marker image. In image 740, the higher the height from the placement surface 600, the darker the color. The image generation unit 106 generates a color image 740 of the image within region 410, which is the region inside the four markers 400, from image 730. For example, the image generation unit 106 divides the image within region 410 into 150 × 150 small regions and calculates the maximum pixel value for each small region. Then, the image generation unit 106 generates image 740 by using this maximum value as the pixel value of each small region. Note that the position of the marker 400 in the corrected depth image can be identified from the color image. Using a similar method, the image generation unit 106 generates a second intra-marker image from the second height image and generates an integrated intra-marker image from the integrated height image.

[0069] It should be noted that how the region 410, which is the region inside the four markers 400, is defined can be adjusted as appropriate. For example, as shown in FIG. 10 , the region connecting the innermost parts of the four markers 400 may be defined as the region 410. Alternatively, the region connecting the outermost parts of the four markers 400 may be defined as the region 410. Also, in FIG. 10 , an example has been described in which the angle formed between the side of the rectangle representing the outer edge of the region 410 and the side of the rectangle representing the outer edge of the image 730 is 0 degrees or 90 degrees. This angle does not have to be 0 degrees or 90 degrees. The image generation unit 106 is an example of an image generation unit.

[0070] The size identification unit 107 identifies the size of the article 500 from the first depth image, the second depth image, and the integrated depth image. Specifically, the size identification unit 107 inputs the first intra-marker image, the second intra-marker image, and the integrated intra-marker image into a trained model, respectively. Then, the size identification unit 107 identifies the size of the article 500 based on the output of the trained model for each of the first intra-marker image, the second intra-marker image, and the integrated intra-marker image.

[0071] The trained model is a model for identifying the size of the article 500 from the in-marker image, and is a model prepared for each type of article 500. The trained model can be generated by learning using training data including an in-marker image standardized to 150 pixels x 150 pixels and the size of the article 500 in real space represented in this in-marker image. The size of the in-marker image is standardized as described above. Therefore, the size of the article 500 in the in-marker image corresponds to the size of the article 500 in real space. For this reason, it is expected that the size of the article 500 can be identified accurately from the in-marker image using the trained model.

[0072] Furthermore, when the color of the placement surface 600 and the color of the article 500 are similar, it is not easy to identify the boundary between the placement surface 600 and the article 500 in an image based on a color image. Even in such a case, it is easy to identify the boundary between the placement surface 600 and the article 500 in an intra-marker image based on a depth image. From this perspective, it is expected that the size of the article 500 can be accurately identified from the intra-marker image using the trained model. In this embodiment, the trained model is created in advance by learning using training data and is stored in the storage unit 12.

[0073] The output data of the trained model can be adjusted as appropriate. For example, the output data of the trained model is data indicating the probability of each of a plurality of size candidates. For example, assume that the length candidates for a certain type of item 500 are 5 cm, 6 cm, and 7 cm. In this case, the output data of the trained model is, for example, data indicating that the probability of 5 cm is 20%, the probability of 6 cm is 60%, and the probability of 7 cm is 20%.

[0074] The size identification unit 107 then comprehensively determines the output data of the trained model for each of the image within the first marker, the image within the second marker, and the image within the integrated marker to identify the size of the article 500. For example, in the above example, the output data when the image within the first marker is input to the trained model is defined as the first output data, the output data when the image within the second marker is input to the trained model is defined as the second output data, and the output data when the image within the integrated marker is input to the trained model is defined as the integrated output data. The first output data is data indicating that there is a 20% probability that the size is 5 cm, a 60% probability that the size is 6 cm, and a 20% probability that the size is 7 cm. The second output data is data indicating that there is a 30% probability that the size is 5 cm, a 50% probability that the size is 6 cm, and a 20% probability that the size is 7 cm. The integrated output data is data indicating that there is a 10% probability that the size is 5 cm, an 85% probability that the size is 6 cm, and a 5% probability that the size is 7 cm.

[0075] In this case, the size identifying unit 107, for example, calculates the average probability of each candidate length being applicable. In the above example, the average probability of 5 cm is 20%, the average probability of 6 cm is 65%, and the average probability of 7 cm is 15%. In this case, the size identifying unit 107 identifies 6 cm, which has the highest average probability of application, as the length of the item 500. The size identifying unit 107 is an example of a size identifying unit.

[0076] The binarization unit 108 binarizes the intra-marker image and generates a binarized image. That is, the binarization unit 108 binarizes the first intra-marker image and generates a first binarized image. The binarization unit 108 also binarizes the second intra-marker image and generates a second binarized image. The binarization unit 108 also binarizes the integrated intra-marker image and generates an integrated binarized image. The binarized image is a general term for the first binarized image, the second binarized image, and the integrated binarized image. How the binarization threshold is set can be adjusted as appropriate.

[0077] For example, a threshold value may be set in advance according to the type of article 500, or a threshold value may be set in advance according to the type of article 500 and the size of the article 500. Alternatively, the threshold value may be set based on the pixel value of each pixel in the intra-marker image. For example, the average value of the pixel values ​​of each pixel in the intra-marker image may be set as the threshold. FIG. 11 shows image 750, which is a first binarized image. Image 750 is an image in which only pixels in image 740 having pixel values ​​equal to or greater than the threshold are expressed in dark colors. The binarization unit 108 is an example of a binarization unit.

[0078] The side identification unit 109 identifies the side of the item 500 from the depth image. Specifically, the side identification unit 109 identifies the side of the item 500 based on the distribution of binarized data in a rectangular intra-image generated based on the depth image. The rectangular intra-image is an image inside a minimum bounding rectangle that surrounds the item 500 in the binarized image. The first rectangular intra-image is an image inside a minimum bounding rectangle that surrounds the item 500 in the first binarized image. The second rectangular intra-image is an image inside a minimum bounding rectangle that surrounds the item 500 in the second binarized image. The integrated rectangular intra-image is an image inside a minimum bounding rectangle that surrounds the item 500 in the integrated binarized image. The rectangular intra-image is a collective term for the first rectangular intra-image, the second rectangular intra-image, and the integrated rectangular intra-image.

[0079] Fig. 11 shows rectangle 751, which is the minimum bounding rectangle surrounding article 500 in image 750, which is the first binarized image. Fig. 12 shows image 760, which is the first intra-rectangle image. Image 760 is the image inside rectangle 751 in image 750. More specifically, image 760 is an image obtained by extracting the image inside rectangle 751 from image 750 and rotating the extracted image so that one side of rectangle 751 is parallel to the vertical axis or horizontal axis of the image.

[0080] The side identification unit 109 identifies a first vector based on the distribution of binarized data in image 760, which is the image within the first rectangle. In FIG. 12, point 761 indicates the center point of image 760, and arrow 762 indicates the first vector. The method by which the side identification unit 109 identifies the first vector can be adjusted as appropriate. In this embodiment, the first vector is identified using principal component analysis. For example, assume that in image 760, the pixel value of the pixel representing the item 500 is 1 and the pixel value of the pixel representing the placement surface 600 is 0. In this case, the side identification unit 109 searches for the axis along which the variance of pixels with a pixel value of 1 is maximized, that is, the axis of the first principal component. The side identification unit 109 identifies a vector extending along this axis as the first vector. Then, the side identification unit 109 identifies the side of the item 500 from the direction in which the first vector extends.

[0081] Hereinafter, with reference to FIG. 13 , a method for identifying the side of the article 500 from the direction in which the first vector extends will be described. The article 500A is a femoral component trial for the left foot, and the article 500B is a femoral component trial for the right foot. The articles 500A and 500B are mirror images of each other. Therefore, when the article 500A is placed on the placement surface 600, the height distribution in the height image differs from when the article 500B is placed on the placement surface 600, regardless of the placement method. This distribution appears in the direction in which the first vector described above extends. Therefore, the side identification unit 109 detects this distribution from the direction in which the first vector extends and identifies the side of the article 500.

[0082] In this embodiment, the side identification unit 109 identifies the side of the article 500 based on the rotation angle when the first vector is rotated clockwise until it first becomes parallel to one of the sides of the minimum bounding rectangle. Specifically, if the rotation angle is 45 degrees or less, the side identification unit 109 identifies the article 500 as a femoral component trial for the left foot, and if the rotation angle is more than 45 degrees, the side identification unit 109 identifies the article 500 as a femoral component trial for the right foot.

[0083] 13, image 760A is an image within a rectangle when article 500A is imaged, and rectangle 751A is the minimum bounding rectangle when article 500A is imaged. Point 761A is the center point of image 760A, and arrow 762A is an arrow indicating a first vector determined from image 760A. Square 763 is a figure for specifying the side of article 500, and point 764 is the center point of square 763. Square 763 is divided into eight regions with point 764 at the center, and different sides are set for adjacent regions.

[0084] Here, image 760A is superimposed on square 763 so that point 761A overlaps point 764 and the sides of rectangle 751A and square 763 are parallel. Next, arrow 762A is rotated clockwise around point 764 until arrow 762A first becomes parallel to one of the sides of square 763. At this time, arrow 762A is positioned in the region L, which represents the left foot, and the rotation angle θa is 45 degrees or less. Therefore, side identification unit 109 determines that item 500A is an item for the left foot.

[0085] 13, image 760B is an image within a rectangle when article 500B is imaged, and rectangle 751B is the minimum bounding rectangle when article 500B is imaged. Point 761B is the center point of image 760B, and arrow 762B is an arrow indicating a first vector determined from image 760B.

[0086] Image 760B is then superimposed on square 763 so that point 761B overlaps point 764 and the sides of rectangle 751B and square 763 are parallel. Arrow 762B is then rotated clockwise around point 764 until arrow 762B first becomes parallel to one of the sides of square 763. At this time, arrow 762B is positioned in the region R, which indicates the right foot, and the rotation angle θb exceeds 45 degrees. Therefore, side identification unit 109 determines that item 500B is an item for the right foot.

[0087] The side identification unit 109 also identifies a second vector based on the distribution of binarized data in the second rectangular image, and identifies the side of the article 500 based on the direction in which the second vector extends. The side identification unit 109 also identifies an integrated vector based on the distribution of binarized data in the integrated rectangular image, and identifies the side of the article 500 based on the direction in which the integrated vector extends. The side identification unit 109 then identifies the side of the article 500 based on the identification result based on the first vector, the identification result based on the second vector, and the identification result based on the integrated vector. For example, the side identification unit 109 identifies the side of the article 500 by majority vote. Specifically, for example, if the identification result based on the first vector is for the left foot, the identification result based on the second vector is for the right foot, and the identification result based on the integrated vector is for the left foot, the side identification unit 109 identifies the side of the article 500 as for the left foot. The side identification unit 109 is an example of a side identification unit.

[0088] The display control unit 110 causes the display unit 13 to display item information based on at least one of the type of item 500 identified by the type identification unit 102, the size of the item 500 identified by the size identification unit 107, and the side of the item 500 identified by the side identification unit 109. The item information may be information indicating at least one of the type of item 500, the size of the item 500, and the side of the item 500, or may be information indicating the result of a determination using at least one of the type of item 500, the size of the item 500, and the side of the item 500. The display control unit 110 is an example of a display control unit.

[0089] The communication control unit 111 transmits the above-mentioned item information to the display device 300 via the second communication unit 16. Meanwhile, the display device 300 displays information based on the item information received from the item identification device 100 on a touch screen, a liquid crystal display, or the like. The processor included in the display device 300, which controls the touch screen, the liquid crystal display, or the like, is an example of a display control unit.

[0090] Next, the item identification process executed by the item identification device 100 will be described with reference to Fig. 14. Note that the item identification process is executed, for example, in response to the operation receiving unit 14 included in the item identification device 100 receiving an operation instructing the start of the item identification process.

[0091] First, the control unit 11 included in the item identification device 100 acquires a color image and a depth image (step S101). Specifically, the control unit 11 acquires a first color image and a first depth image from the imaging device 200A, and acquires a second color image and a second depth image from the imaging device 200B. After completing the process of step S101, the control unit 11 identifies the type of the item 500 from the color image (step S102). For example, the control unit 11 identifies the type of the item 500 from the first color image using a known pattern matching technique, a known machine learning technique, or the like.

[0092] When the process of step S102 is completed, the control unit 11 executes a size specification process (step S103). The size specification process will be described in detail with reference to FIG.

[0093] First, the control unit 11 corrects the depth image into a corrected depth image (step S201). Specifically, the control unit 11 corrects the first depth image into a first corrected depth image, and corrects the second depth image into a second corrected depth image. After completing the process of step S201, the control unit 11 generates an integrated corrected depth image (step S202). That is, the control unit 11 generates an integrated corrected depth image from the first corrected depth image and the second corrected depth image.

[0094] After completing the process of step S202, the control unit 11 selects a corrected depth image (step S203). That is, the control unit 11 selects one corrected depth image from three corrected depth images, namely, the first corrected depth image, the second corrected depth image, and the integrated corrected depth image. After completing the process of step S203, the control unit 11 converts the corrected depth image into a height image (step S204). For example, the control unit 11 converts the selected first corrected depth image into a first height image.

[0095] Upon completing the process of step S204, the control unit 11 generates an intra-marker image (step S205). For example, the control unit 11 generates a first intra-marker image from the first height image. Upon completing the process of step S205, the control unit 11 inputs the intra-marker image to the trained model (step S206). For example, the control unit 11 inputs the first intra-marker image to the trained model.

[0096] After completing the process of step S206, the control unit 11 acquires output data of the trained model (step S207). For example, the control unit 11 acquires output data indicating the probability of each candidate size of the article 500 from the trained model. After completing the process of step S207, the control unit 11 determines whether or not there is an unselected corrected depth image (step S208). That is, the control unit 11 determines whether or not all three corrected depth images have been selected.

[0097] If the control unit 11 determines that there is an unselected corrected depth image (step S208: YES), the control unit 11 returns the process to step S203. In this case, the control unit 11 selects a new corrected depth image in step S203 and executes the processes of steps S204 to S208. If the control unit 11 determines that there is no unselected corrected depth image (step S208: NO), the control unit 11 identifies the size of the article 500 from each output data (step S209). That is, the control unit 11 identifies the size of the article 500 based on the first output data, the second output data, and the integrated output data. When the control unit 11 completes the process of step S209, the size identification process is completed.

[0098] When the control unit 11 completes the size specification process in step S103, it executes a side specification process (step S104). The side specification process will be described in detail with reference to FIG.

[0099] First, the control unit 11 selects an intra-marker image (step S301). That is, the control unit 11 selects one intra-marker image from three intra-marker images, namely, the first intra-marker image, the second intra-marker image, and the integrated intra-marker image. After completing the process of step S301, the control unit 11 generates a binarized image from the intra-marker image (step S302). For example, the control unit 11 generates a first binarized image from the selected first intra-marker image.

[0100] Upon completing the process of step S302, the control unit 11 generates a rectangular image from the binarized image (step S303). For example, the control unit 11 generates a first rectangular image from the first binarized image. Upon completing the process of step S303, the control unit 11 identifies a vector from the distribution of the binary data (step S304). For example, the control unit 11 identifies a first vector from the distribution of the binary data indicated by the first rectangular image.

[0101] Upon completing the process of step S304, the control unit 11 identifies the side of the article 500 from the direction in which the vector extends (step S305). For example, the control unit 11 identifies the side of the article 500 from the direction in which the first vector extends. Upon completing the process of step S305, the control unit 11 determines whether or not there is an unselected image within the marker (step S306). In other words, the control unit 11 determines whether or not all three images within the marker have been selected.

[0102] If the control unit 11 determines that there is an unselected intra-marker image (step S306: YES), the control unit 11 returns the process to step S301. In this case, the control unit 11 selects a new intra-marker image in step S301 and executes the processes of steps S302 to S306. If the control unit 11 determines that there is no unselected intra-marker image (step S306: NO), the control unit 11 identifies the side of the article 500 by majority vote (step S307). When the control unit 11 completes the process of step S307, the side identification process is completed.

[0103] Upon completing the side identification process of step S104, the control unit 11 displays the item information (step S105). For example, the control unit 11 displays item information related to at least one of the type of item 500, the size of the item 500, and the side of the item 500 on the display unit 13. Upon completing the process of step S105, the control unit 11 transmits the item information to the display device 300 (step S106). For example, the control unit 11 transmits the item information to the display device 300 via the second communication unit 16. Upon completing the process of step S106, the control unit 11 completes the item identification process.

[0104] As described above, in this embodiment, the size of the article 500 is identified from the depth image. Therefore, according to this embodiment, the size of the article 500 can be identified with high accuracy. Furthermore, in this embodiment, the size of the article 500 is identified based on output data of a trained model to which an intra-marker image is input. Therefore, according to this embodiment, the size of the article 500 can be identified with even higher accuracy. Furthermore, in this embodiment, the size of the article is identified based on output data of a trained model for each of the first intra-marker image, the second intra-marker image, and the integrated intra-marker image. Therefore, according to this embodiment, the size of the article 500 can be identified with even higher accuracy.

[0105] In this embodiment, the side of the article 500 is identified from the depth image. Therefore, according to this embodiment, the side of the article 500 can be identified with high accuracy. Furthermore, in this embodiment, the side of the article 500 is identified based on the distribution of binarized data in the rectangular image. Therefore, according to this embodiment, the side of the article 500 can be identified with even higher accuracy. Furthermore, in this embodiment, the side of the article 500 is identified based on the distribution of binarized data in the first rectangular image, the distribution of binarized data in the second rectangular image, and the distribution of binarized data in the integrated rectangular image. Therefore, according to this embodiment, the side of the article 500 can be identified with even higher accuracy.

[0106] (Variation) Although the embodiments of the present disclosure have been described above, modifications and applications in various forms are possible.

[0107] It is up to the discretion of which parts of the configurations, functions, and operations described in the above embodiments to adopt. Furthermore, in addition to the above-described configurations, functions, and operations, further configurations, functions, and operations may be adopted. Furthermore, the above-described embodiments can be freely combined as appropriate. Furthermore, the number of components described in the above-described embodiments can be adjusted as appropriate. Furthermore, it goes without saying that the materials, sizes, electrical characteristics, and the like that can be adopted are not limited to those shown in the above-described embodiments.

[0108] In the embodiment, an example has been described in which four markers 400 are arranged at positions corresponding to the vertices of a square. As long as the markers 400 can identify the scale of the captured image and can be used to align two captured images captured by two imaging devices 200, the number and arrangement of the markers 400 can be adjusted as appropriate. For example, four markers 400 may be arranged at positions corresponding to the vertices of a rectangle. Alternatively, for example, three markers 400 may be arranged at positions corresponding to the vertices of an equilateral triangle.

[0109] In the embodiment, an example has been described in which the size of the article 500 is identified based on the first intra-marker image, the second intra-marker image, and the integrated intra-marker image. The size of the article 500 may be identified based on the first intra-marker image, the size of the article 500 may be identified based on the second intra-marker image, or the size of the article 500 may be identified based on the integrated intra-marker image.

[0110] Furthermore, in the embodiment, an example has been described in which the side of the article 500 is identified based on the first rectangular image, the second rectangular image, and the integrated rectangular image. The side of the article 500 may be identified based on the first rectangular image, the side of the article 500 may be identified based on the second rectangular image, or the side of the article 500 may be identified based on the integrated rectangular image.

[0111] In the embodiment, an example has been described in which the number of imaging devices 200 is two. The number of imaging devices 200 may be one, or three or more. When the number of imaging devices 200 is one, a corrected depth image, an intra-marker image, a binarized image, and an intra-rectangle image are generated from the depth image, the size of the article 500 is identified based on the intra-marker image, and the sides of the article 500 are identified based on the intra-rectangle image.

[0112] When the number of imaging devices 200 is three or more, three or more corrected depth images are generated from the three or more depth images, and one or more integrated corrected depth images are generated from the three or more corrected depth images. For example, when three corrected depth images exist, there are three combinations including two corrected depth images, and one combination including three corrected depth images. Therefore, a maximum of four integrated corrected depth images are generated from the three corrected depth images. Then, a plurality of intra-marker images including at least one integrated intra-marker image, a plurality of binarized images including at least one integrated binarized image, and a plurality of intra-rectangle images including at least one integrated intra-rectangle image are generated. Then, the size of the article 500 is identified based on the plurality of intra-marker images, and the sides of the article 500 are identified based on the plurality of intra-rectangle images.

[0113] In the embodiment, an example has been described in which one imaging device 200 is equipped with an image sensor and a depth sensor and generates a color image and a depth image. An imaging device 200 equipped with an image sensor and generating a color image, and an imaging device 200 equipped with a depth sensor and generating a depth image may be provided. In this case, it is sufficient that the imaging device 200 equipped with the image sensor and the imaging device 200 equipped with the depth sensor are located at approximately the same location, and that the imaging areas of the imaging device 200 equipped with the image sensor and the imaging device 200 equipped with the depth sensor are approximately the same.

[0114] In the embodiment, an example has been described in which the item identification device 100 identifies the size and the side of the item 500. The item identification device 100 may identify the side of the item 500 without identifying the size of the item 500, or may identify the size of the item 500 without identifying the side of the item 500.

[0115] In the embodiment, an example has been described in which the item identification system 1000 includes the item identification device 100, the imaging device 200, and the display device 300. The item identification system 1000 may have any configuration as long as it has the functions described in the embodiment as a whole. For example, the item identification system 1000 may include the item identification device 100 and the imaging device 200, and the item identification device 100 may have the functions of the display device 300. Alternatively, the item identification system 1000 may include the item identification device 100 and the imaging device 200, and the imaging device 200 may have the functions of the display device 300.

[0116] In this case, the imaging device 200 is preferably goggles capable of realizing AR (Augmented Reality) or MR (Mixed Reality). In this case, for example, a nurse who delivers instruments wears the goggles, and the goggles capture images of the multiple markers 400, the objects 500, and the placement surface 600 to generate color images and depth images. The object identification device 100 acquires the color image and the depth image from the goggles, identifies the type of the object 500 from the color image, and identifies the size and side of the object 500 from the depth image. The goggles acquire object information including the type, size, and side of the object 500 from the object identification device 100. The goggles can display information including the type, size, and side of the object 500, etc., superimposed on the image captured by the goggles.

[0117] Furthermore, the item identification device 100 may determine whether the type, size, or side of the item 500 is predetermined, and provide item information indicating the determination result to the goggles. In this case, the goggles may display information indicating the determination result superimposed on an image captured by the goggles. Furthermore, when the goggles capture an image of multiple items 500, the item identification device 100 may determine whether the multiple items 500 correspond to each other in terms of type, size, side, or the like. In this case, the goggles may display information indicating the determination result superimposed on an image captured by the goggles. Note that a corresponding relationship refers to a relationship of compatibility, a relationship of simultaneous use, or the like.

[0118] In the above embodiment, the CPU of the control unit 11 executes a program stored in the ROM or the storage unit 12, thereby functioning as each unit shown in FIG. 4 . However, in the present disclosure, the control unit 11 may be dedicated hardware. Dedicated hardware may be, for example, a single circuit, a composite circuit, a programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. When the control unit 11 is dedicated hardware, the functions of each unit may be realized by individual hardware, or the functions of each unit may be realized collectively by a single piece of hardware. Furthermore, some of the functions of each unit may be realized by dedicated hardware, and the other functions may be realized by software or firmware. In this way, the control unit 11 can realize each of the above-described functions by hardware, software, firmware, or a combination thereof.

[0119] By applying an operation program that defines the operation of the item identification device 100 according to the present disclosure to a computer such as an existing personal computer or information terminal device, it is also possible to cause the computer to function as the item identification device 100 according to the present disclosure. In addition, the method of distribution of such a program is arbitrary, and for example, the program may be stored and distributed on a computer-readable recording medium such as a CD-ROM (Compact Disk ROM), a DVD (Digital Versatile Disk), an MO (Magneto Optical Disk), or a memory card, or may be distributed via a communication network such as the Internet.

[0120] The present disclosure allows various embodiments and modifications without departing from the broad spirit and scope of the present disclosure. Furthermore, the above-described embodiments are intended to explain the present disclosure and do not limit the scope of the present disclosure. That is, the scope of the present disclosure is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of equivalent disclosures are considered to be within the scope of the present disclosure. [Explanation of symbols]

[0121] 11 control unit, 12 memory unit, 13 display unit, 14 operation reception unit, 15 first communication unit, 16 second communication unit, 71, 75, 76 point, 72 plane, 73 line, 74, 762, 762A, 762B arrow, 100 item identification device, 101 image acquisition unit, 102 type identification unit, 103 image correction unit, 104 image integration unit, 105 image conversion unit, 106 image generation unit, 107 size identification unit, 108 binarization unit, 109 side identification unit, 110 display control unit, 111 communication control unit, 200, 200A, 200B imaging device, 211, 212, 761, 761A, 761B, 764 point, 251, 252 support member, 300 Display device, 400, 400A, 400B, 400C, 400D Marker, 410 Area, 500, 500A, 500B Item, 600 Placement surface, 710, 720, 730, 740, 750, 760, 760A, 760B Image, 751, 751A, 751B Rectangle, 763 Square, 1000 Item identification system

Claims

1. an image acquisition unit that acquires a first color image and a first depth image by capturing images of a plurality of markers, an object placed between the plurality of markers, and a surface on which the plurality of markers and the object are placed from a first point; a type identification unit that identifies the type of the article from the first color image; a size identification unit that identifies a size of the item from the first depth image; an image corrector that corrects the first depth image into a first corrected depth image that represents a depth when the placement surface is viewed from a direction perpendicular to the placement surface; an image conversion unit that converts the first corrected depth image into a first height image that represents a height from the placement surface; an image generating unit that generates a first intra-marker image that is an image of an area inside the plurality of markers in the first height image, The size identification unit inputs the image within the first marker into a trained model and identifies the size of the item based on output data of the trained model. Article identification device.

2. the image acquisition unit acquires a second color image and a second depth image obtained by capturing an image of the plurality of markers, the item, and the placement surface from a second point; the image correcting unit corrects the second depth image to a second corrected depth image that represents a depth when the placement surface is viewed from a direction perpendicular to the placement surface; the image conversion unit converts the second corrected depth image into a second height image representing a height from the placement surface; the image generation unit generates a second intra-marker image, which is an image of an area inside the plurality of markers in the second height image; the size identification unit inputs the second intra-marker image to the trained model, and identifies the size of the article based on output data of the trained model for each of the first intra-marker image and the second intra-marker image. The article identification device according to claim 1 .

3. an image integration unit that generates an integrated corrected depth image by integrating the first corrected depth image and the second corrected depth image; the image conversion unit converts the integrated corrected depth image into an integrated height image representing a height from the placement surface; the image generation unit generates an integrated intra-marker image, which is an image of an area inside the plurality of markers in the integrated height image; the size identification unit inputs the integrated intra-marker image to the trained model, and identifies the size of the item based on output data of the trained model for each of the first intra-marker image, the second intra-marker image, and the integrated intra-marker image. The article identification device according to claim 2 .

4. An image acquisition unit that acquires a first color image and a first depth image obtained by capturing images of a plurality of markers, an item placed between the plurality of markers, and a placement surface on which the plurality of markers and the item are placed from a first point; a type identification unit that identifies the type of the article from the first color image; a size identification unit that identifies the size of the item from the first depth image, The item is identified by type, size, and a side that designates one of a pair of mirror-image items; a side identification unit that identifies a side of the article from the first depth image; Article identification device.

5. an image corrector that corrects the first depth image into a first corrected depth image that represents a depth when the placement surface is viewed from a direction perpendicular to the placement surface; an image conversion unit that converts the first corrected depth image into a first height image that represents a height from the placement surface; an image generating unit that generates a first intra-marker image, which is an image of an area inside the plurality of markers in the first height image; a binarization unit that generates a first binarized image by binarizing the first intra-marker image, the side identification unit identifies the side of the article based on a distribution of binarized data in a first intra-rectangle image, which is an image inside a minimum bounding rectangle that surrounds the article in the first binarized image. The article identification device according to claim 4 .

6. the image acquisition unit acquires a second color image and a second depth image obtained by capturing an image of the plurality of markers, the item, and the placement surface from a second point; the image correcting unit corrects the second depth image to a second corrected depth image that represents a depth when the placement surface is viewed from a direction perpendicular to the placement surface; the image conversion unit converts the second corrected depth image into a second height image representing a height from the placement surface; the image generation unit generates a second intra-marker image, which is an image of an area inside the plurality of markers in the second height image; the binarization unit generates a second binarized image by binarizing the second intra-marker image; the side identification unit identifies a side of the article based on a distribution of binarized data in the first rectangular image and a distribution of binarized data in a second rectangular image, which is an image inside a minimum bounding rectangle that surrounds the article in the second binarized image. The article identification device according to claim 5 .

7. an image integration unit that generates an integrated corrected depth image by integrating the first corrected depth image and the second corrected depth image; the image conversion unit converts the integrated corrected depth image into an integrated height image representing a height from the placement surface; the image generation unit generates an integrated intra-marker image, which is an image of an area inside the plurality of markers in the integrated height image; the binarization unit generates an integrated binarized image by binarizing the image within the integrated marker; the side identification unit identifies the side of the article based on a distribution of binary data in the first rectangular image, a distribution of binary data in the second rectangular image, and a distribution of binary data in an integrated rectangular image, which is an image inside a minimum bounding rectangle that surrounds the article in the integrated binary image. The article identification device according to claim 6 .

8. an imaging unit that captures images of a plurality of markers, an object placed between the plurality of markers, and a surface on which the plurality of markers and the object are placed from a first point, and generates a first color image and a first depth image; a type identification unit that identifies the type of the article from the first color image; a size identification unit that identifies a size of the item from the first depth image; a display control unit that causes a display unit to display product information based on at least one of the type of product identified by the type identification unit and the size of the product identified by the size identification unit; and an image corrector that corrects the first depth image into a first corrected depth image that represents a depth when the placement surface is viewed from a direction perpendicular to the placement surface; an image conversion unit that converts the first corrected depth image into a first height image that represents a height from the placement surface; an image generating unit that generates a first intra-marker image that is an image of an area inside the plurality of markers in the first height image, The size identification unit inputs the image within the first marker into a trained model and identifies the size of the item based on output data of the trained model. Item identification system.

9. An imaging unit that captures images of a plurality of markers, an item placed between the plurality of markers, and a placement surface on which the plurality of markers and the item are placed from a first point, and generates a first color image and a first depth image; a type identification unit that identifies the type of the article from the first color image; a size identification unit that identifies a size of the item from the first depth image; a display control unit that causes a display unit to display product information based on at least one of the product type identified by the product type identification unit and the product size identified by the product size identification unit; The item is identified by type, size, and a side that designates one of a pair of mirror-image items; a side identification unit that identifies a side of the article from the first depth image; Item identification system.

10. Computer, an image acquisition unit that acquires a first color image and a first depth image by capturing images of a plurality of markers, an object placed between the plurality of markers, and a surface on which the plurality of markers and the object are placed from a first point; a type identification unit that identifies the type of the article from the first color image; a size identification unit that identifies the size of the item from the first depth image; an image correcting unit that corrects the first depth image into a first corrected depth image that represents a depth when the placement surface is viewed from a direction perpendicular to the placement surface; an image conversion unit that converts the first corrected depth image into a first height image that represents a height from the placement surface; a program that functions as an image generating unit that generates a first intra-marker image that is an image of an area inside the plurality of markers in the first height image, The size identification unit inputs the image within the first marker into a trained model and identifies the size of the item based on output data of the trained model. program.

11. A computer, an image acquisition unit that acquires a first color image and a first depth image by capturing images of a plurality of markers, an object placed between the plurality of markers, and a surface on which the plurality of markers and the object are placed from a first point; a type identification unit that identifies the type of the article from the first color image; a program that functions as a size identification unit that identifies the size of the item from the first depth image, The item is identified by type, size, and a side that designates one of a pair of mirror-image items; The program causes the computer to: and further functioning as a side identification unit that identifies a side of the article from the first depth image. program.

Images