Image acquisition device for 3D model generation

Abstract

To generate a further precise 3D model with a reduced effort, when generating a 3D model from a captured image obtained by capturing work environment of a robot.SOLUTION: An image acquisition device for 3D model generation comprises: a captured image acquisition unit 11 which accepts input of a captured image; a marker image acquisition unit 12 which accepts input of a marker image which is an image of a marker used as a reference of a position when generating a 3D model; a composition destination extraction unit 13 which extracts a composition destination region which is a region of a composition destination of the marker image, from the captured image acquired by the captured image acquisition unit 11; an image composition unit 14 which synthesizes the marker image accepted as the input in the marker image acquisition unit 12, to the composition destination region extracted by the composition destination extraction unit 13, among the captured images; and an image output unit 15 which outputs a composite image which is an image obtained by synthesizing the marker image to the captured image in the image composition unit 14.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image acquisition device for generating a 3D model. [Background technology]

[0002] Labor shortages in production sites due to a decline in the working-age population have become a social issue, creating a demand for automated systems that use robots to work in place of humans. However, it takes a great deal of effort to position robots in real space so that they can perform the desired operations while avoiding interference with surrounding structures. Therefore, when introducing robots into production sites, it is desirable to reduce the integration costs required for system integration. In response to this, for example, Patent Document 1 discloses a technology for constructing a virtual robot system in which a virtual three-dimensional model of a robot and a structure surrounding the robot is arranged in a virtual space, and teaching the robot a movement path. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-140958 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, even with the technology disclosed in Patent Document 1, there is a problem in that integration costs become high if there is a discrepancy between the virtual three-dimensional model (hereinafter referred to as the 3D model) and the actual operating environment of the robot. This is because such a discrepancy requires time and effort to operate the actual robot and redo the teaching while checking whether the movements are actually performed as instructed. Therefore, in order to eliminate the discrepancy between the 3D model and the actual operating environment and reduce integration costs, it is necessary to generate a precise 3D model of the robot's working environment.

[0005] When generating a 3D model from captured images of a robot's working environment, it is possible to generate a precise 3D model that minimizes deviations from the actual operating environment by placing markers in the working environment in advance as positional references. However, placing markers in the working environment requires a lot of effort to select, install, and retrieve the markers. In particular, when continuously creating 3D models of a site such as a factory where equipment is frequently rearranged, it is necessary to reduce the effort required to operate the markers. Furthermore, if the markers used to generate the 3D model are changed due to a change in the software used to generate the 3D model, it becomes time-consuming to install markers compatible with the new software.

[0006] One objective of this disclosure is to provide an image acquisition device for generating a 3D model that enables the generation of a more precise 3D model with less effort when generating a 3D model from captured images of a robot's working environment. [Means for solving the problem]

[0007] The above object is achieved by the combination of features recited in the independent claims, and the subclaims define further advantageous embodiments of the disclosure. The reference numerals in parentheses in the claims correspond to specific means described in the following embodiment as one aspect, and do not limit the technical scope of the present disclosure.

[0008] In order to achieve the above object, the image acquisition device for generating a 3D model of the present disclosure is an image acquisition device for generating a 3D model that acquires captured images of an operating environment for use in generating a 3D model of the operating environment of a robot, and includes: a captured image acquisition unit (11) that accepts input of the captured image; a marker image acquisition unit (12, 12a) that accepts input of a marker image, which is an image of a marker used as a positional reference when generating a 3D model; a synthesis destination extraction unit (13, 13a) that extracts a synthesis destination area, which is an area to which the marker image is to be synthesized, from the captured image acquired by the captured image acquisition unit; an image synthesis unit (14) that synthesizes the marker image accepted as input by the marker image acquisition unit with the synthesis destination area extracted by the synthesis destination extraction unit of the captured image; and an image output unit (15) that outputs a synthesized image, which is an image obtained by synthesizing the marker image with the captured image by the image synthesis unit.

[0009] According to the above configuration, it is possible to output a composite image in which marker images of markers used as positional references when generating a 3D model are superimposed on an area in a captured image of the robot's operating environment. Therefore, even without installing markers in the actual operating environment in advance, it is possible to generate a precise 3D model with reduced deviation from the actual operating environment from the composite image in which the marker images are superimposed. Furthermore, even if the markers used to generate the 3D model change due to a change in the software used to generate the 3D model, this can be easily accommodated by simply changing the marker images. This eliminates the need to re-install markers compatible with the new software in the actual operating environment. Therefore, even when continuously creating 3D models of a factory, it is possible to reduce the effort required to operate markers. As a result, when generating a 3D model from captured images of the robot's working environment, it is possible to generate a more precise 3D model with less effort. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of an image acquisition device for generating a 3D model according to a first embodiment. [Figure 2]10A and 10B are diagrams illustrating an example of combining a marker image with a combination destination area in a captured image. [Figure 3] 10 is a flowchart showing an example of image synthesis-related processing when CAD data is used in an image acquisition device for generating a 3D model. [Figure 4] 10 is a flowchart showing an example of image synthesis-related processing in an image acquisition device for generating a 3D model when CAD data is not used. [Figure 5] FIG. 10 is a diagram illustrating an example of a schematic configuration of an image acquisition device for generating a 3D model according to a second embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of a correspondence relationship between an address and a marker image. [Figure 7] FIG. 10 is a diagram showing an example of a captured image of an operating environment to which a two-dimensional code is attached. [Figure 8] FIG. 10 is a diagram illustrating an example of an image for generating a model. DETAILED DESCRIPTION OF THE INVENTION

[0011] A number of embodiments for the purpose of disclosure will be described with reference to the drawings. For the sake of convenience, parts having the same functions as parts shown in the drawings used in the previous explanations in the number of embodiments will be given the same reference numerals, and their description may be omitted. For parts given the same reference numerals, the explanations in other embodiments may be referred to.

[0012] (Embodiment 1) <Schematic configuration of image acquisition device 1 for generating 3D model> A first embodiment of the present disclosure will be described below with reference to the drawings. The image acquisition device 1 for 3D model generation shown in FIG. 1 provides captured images used to generate a 3D model of a robot's operating environment. The operating environment may, for example, be the robot and its surrounding structures. Hereinafter, the robot's operating environment will be simply referred to as the operating environment. Hereinafter, captured images used to generate a 3D model of the robot's operating environment will be referred to as images for model generation. Although the operating environment does not necessarily include the robot, the following description will continue assuming that the operating environment includes the robot. The image acquisition device 1 for 3D model generation is primarily composed of a computer including, for example, a processor, volatile memory, nonvolatile memory, I / O, and buses connecting these. The image acquisition device 1 for 3D model generation executes a control program stored in the nonvolatile memory to perform processing related to providing images for model generation.

[0013] Next, the schematic configuration of the image acquisition device 1 for generating a 3D model will be described with reference to Fig. 1. As shown in Fig. 1, the image acquisition device 1 for generating a 3D model includes functional blocks: a captured image acquisition unit 11, a marker image acquisition unit 12, a synthesis destination extraction unit 13, an image synthesis unit 14, and an image output unit 15. Note that some or all of the functions executed by the image acquisition device 1 for generating a 3D model may be configured as hardware using one or more ICs or the like. Furthermore, some or all of the functional blocks included in the image acquisition device 1 for generating a 3D model may be realized by a combination of software executed by a processor and hardware components.

[0014] The captured image acquisition unit 11 accepts input of captured images of the operating environment. The captured images to be accepted are preferably multiple captured images of the same operating environment captured from multiple directions. This is because, when generating a 3D model using captured images of an object, it is possible to generate a 3D model with higher accuracy. Methods for generating a 3D model using multiple captured images of the same object captured from multiple directions include SfM (Structure from Motion), 3D Gaussian Splatting, and Neural Radiance Fields (NeRF). In this embodiment, the description will continue with an example in which a 3D model is generated from captured images using SfM. The captured images may be input in any type of input. For example, they may be input via a network. Alternatively, they may be input from a storage medium such as a memory card that is insertable into or removable from the image acquisition device 1 for 3D model generation.

[0015] The marker image acquisition unit 12 accepts input of an image of a marker (hereinafter, a marker image) used as a positional reference when generating a 3D model. A marker used as a positional reference when generating a 3D model is hereinafter referred to as a reference marker. The reference marker is used as a positional reference to improve the accuracy of the correspondence between a position in real space and a position in virtual space. The marker image accepted as input by the marker image acquisition unit 12 is a marker image recognized as a reference marker by software that generates a 3D model using an image for model generation. When the software for generating a 3D model is changed and the type of recognizable reference marker is changed, the marker image acquisition unit 12 simply accepts input of a marker image corresponding to this change. The marker image may be input in any type of input. For example, it may be input via a network. Alternatively, it may be input from a storage medium such as a memory card that is insertable into or removable from the image acquisition device 1 for 3D model generation.

[0016] It is preferable that the marker image acquisition unit 12 acquires multiple types of marker images that are distinguishable from one another as the marker images. In other words, it is preferable to acquire multiple types of unique marker images. When generating a 3D model using model generation images, if there are multiple reference markers, the reference markers are more likely to be included in images captured from more imaging directions. Therefore, with the above configuration, it becomes possible to generate a 3D model from images captured from more imaging directions, making it easier to generate a precise 3D model.

[0017] The synthesis destination extraction unit 13 extracts a synthesis destination area for the marker image (hereinafter referred to as synthesis destination area) from the captured image acquired by the captured image acquisition unit 11. The synthesis destination extraction unit 13 extracts an area from the captured image that meets the conditions for the synthesis destination area. The synthesis destination extraction unit 13 may extract an area from the captured image that meets the conditions for the synthesis destination area using image recognition technology. The condition for the synthesis destination area is that an area that is likely to be unique within the operating environment must be easily distinguished and recognized using image recognition technology. When the marker image acquisition unit 12 acquires multiple types of marker images that are distinguishable from one another, the synthesis destination extraction unit 13 may do the following. The synthesis destination extraction unit 13 may extract multiple distinguishable areas as synthesis destination areas. Details of the extraction of the synthesis destination area will be described later.

[0018] The image composition unit 14 composites the marker image acquired by the marker image acquisition unit 12 into the composite destination area extracted by the composite destination extraction unit 13 in the captured image acquired by the captured image acquisition unit 11. In this way, an image for model generation is generated in which the marker image is composited into the captured image. The marker image may be composited so that the reference marker is actually positioned in the composite destination area by performing image conversion (see FIG. 2) in accordance with the inclination of the surface of the composite destination area in the captured image. FIG. 2 is a diagram for explaining an example of compositing the marker image into the composite destination area in the captured image. FI in FIG. 2 indicates the captured image. CA in FIG. 2 indicates the composite area. MIB in FIG. 2 indicates the marker image before image conversion. MIA in FIG. 2 indicates the marker image after image conversion. When the marker image acquisition unit 12 acquires multiple types of marker images that are distinguishable from one another, the image composition unit 14 may do the following. The image composition unit 14 may composite the multiple types of marker images acquired by the marker image acquisition unit 12 into the multiple composite destination areas extracted by the composite destination extraction unit 13 in the captured image, respectively. This makes it possible to generate a 3D model from images captured in more imaging directions, making it easier to generate a precise 3D model.

[0019] The image output unit 15 outputs the image for model generation as a composite image obtained by combining the marker image with the captured image by the image composition unit 14. The image output unit 15 may output the image for model generation via a network to a device that generates a 3D model from the captured image. The device that generates a 3D model from the captured image is hereinafter referred to as a 3D model generation device. The image output unit 15 may output the image for model generation to a storage medium that can be inserted into or removed from the image acquisition device for 3D model generation 1. Note that the 3D model generation device to which this storage medium is connected may accept input of the image for model generation from the storage medium and generate a 3D model using the image for model generation.

[0020] According to the above configuration, it is possible to output a composite image in which a marker image of a fiducial marker is superimposed on an area in a captured image of the operating environment. Therefore, even without installing fiducial markers in the actual operating environment in advance, it is possible to generate a precise 3D model with reduced deviation from the actual operating environment from the composite image in which the marker image is superimposed. Furthermore, even if the fiducial markers used to generate the 3D model are changed due to a change in the software used to generate the 3D model, this can be easily accommodated by changing the marker image. This eliminates the need to re-install fiducial markers in the actual operating environment to accommodate the new software. Therefore, even when continuously creating 3D models of a factory, it is possible to reduce the effort required to operate the fiducial markers. As a result, when generating a 3D model from captured images of a robot's working environment, it is possible to generate a more precise 3D model with less effort.

[0021] A 3D model generation device generates a 3D model using images for model generation, for example, using SfM. Here, the flow of generating a 3D model using images for model generation using SfM will be described. First, the 3D model generation device recognizes fiducial markers from the images for model generation using image recognition technology. In SfM, images for model generation can be generated from multiple images captured from multiple directions of the same operating environment. The 3D model generation device estimates the shooting position for each of the multiple images for model generation. Then, the 3D model generation device constructs point cloud data for the entire operating environment from the parallax of each image for model generation relative to the same fiducial marker, and generates a 3D model of the operating environment.

[0022] Next, the extraction of the synthesis destination area will be described in detail. When design data of the operating environment is available, the synthesis destination extraction unit 13 may do the following. Using the design data of the operating environment, the synthesis destination extraction unit 13 may extract, as the synthesis destination area, an area that meets the conditions of the synthesis destination area from the captured image. The design data may be, for example, CAD data of the operating environment. The following description will continue taking as an example a case where the design data is CAD data. The image acquisition device 1 for generating a 3D model may acquire the CAD data via a network or via a removable storage medium.

[0023] When CAD data is available, the composite destination extraction unit 13 may select an area that meets the conditions for the composite destination area from a CAD model generated from the CAD data. Then, the composite destination extraction unit 13 may extract an area that is recognized as the same as the selected area as the composite destination area from the captured image acquired by the captured image acquisition unit 11 using image recognition technology. In this way, by using design data of the operating environment, the composite destination area can be more easily extracted than when this design data is not used. In addition, deviations and failures in the extraction of the composite destination area are less likely to occur. When extracting multiple areas that are distinguishable from each other, the composite destination extraction unit 13 may extract areas with different characteristics as the composite destination areas.

[0024] The composite destination extraction unit 13 preferably selects a region of a predetermined shape with a predetermined aspect ratio from the CAD model as a region that meets the conditions for the composite destination region. In other words, the composite destination extraction unit 13 preferably extracts a region of a predetermined shape with a predetermined aspect ratio as a region that meets the conditions for the composite destination region. The "predetermined shape" referred to here can be set arbitrarily and can be a shape that is easily recognized as an image and is likely to have a wide surface. For example, it can be a rectangle, a circle, or the like. The "predetermined aspect ratio" can be set arbitrarily and can be a ratio that corresponds to a region that is likely to be unique in the operating environment. One example is the aspect ratio of a teaching pendant for a robot in the operating environment. The condition that the aspect ratio is a predetermined ratio is hereinafter referred to as the first condition.

[0025] According to the above configuration, by using the design data of the operating environment, it becomes easier to extract a unique area within the operating environment as a synthesis destination area. Therefore, when extracting synthesis destination areas in different captured images, it becomes less likely that different areas in each captured image will be extracted as the same synthesis destination area. Therefore, it becomes less likely that the same marker image will be spun into different areas in each captured image. As a result, it becomes possible to provide model generation images that enable the generation of more precise 3D models. Note that when extracting multiple regions that are distinguishable from one another, the synthesis destination extraction unit 13 may extract regions with different aspect ratios or shapes as synthesis destination areas.

[0026] The synthesis target extraction unit 13 preferably selects a region of a predetermined shape containing a predetermined identifier from the CAD model as a region that meets the synthesis target region conditions. In other words, the synthesis target extraction unit 13 preferably extracts a synthesis target region by determining a region of a predetermined shape containing a predetermined identifier as a region that meets the synthesis target region conditions. The predetermined shape referred to here may be the same as described above. The predetermined identifier may be arbitrarily set and may be an identifier that is likely to be unique within the operating environment. Examples include manufacturer logos, two-dimensional codes, SfM markers, and character strings attached to equipment included in the operating environment. SfM markers are markers used as positional references in SfM. The significance of synthesizing a marker image in a region already marked with an SfM marker is that it enables the generation of a 3D model using the marker image even if the existing SfM markers become unusable for generating a 3D model due to a software change. Examples of character strings include equipment operating instructions and cautions. The condition of including a predetermined identifier is hereinafter referred to as the second condition. The above configuration makes it easier to extract a unique area within the operating environment as a composite-destination area. As a result, it becomes possible to provide a model generation image that enables the generation of a more precise 3D model. Note that when extracting multiple areas that are distinguishable from one another, the composite-destination extraction unit 13 may extract areas with different identifiers as composite-destination areas.

[0027] Preferably, the composite destination extraction unit 13 selects a region of a predetermined shape that satisfies the second condition in addition to the first condition as a region that meets the conditions for the composite destination region. In this way, even if the first condition alone is not enough to narrow down the region to a unique region within the operating environment, adding the second condition also makes it easier to narrow down the region to a unique region. As a result, it becomes possible to provide a model generation image that can generate a more precise 3D model.

[0028] The composite-destination extraction unit 13 preferably selects a region of a predetermined shape containing a predetermined percentage of a predetermined color from the CAD model as a region that meets the conditions for the composite-destination region. In other words, the composite-destination extraction unit 13 preferably extracts a region of a predetermined shape containing a predetermined percentage of a predetermined color as a region that meets the conditions for the composite-destination region. The predetermined shape referred to here may be the same as described above. The predetermined percentage of the predetermined color may be set arbitrarily and may be a color and its percentage that are characteristic of a region that is likely to be unique within the operating environment. For example, it may be a color and its percentage that are characteristic of a warning sticker attached to equipment included in the operating environment. The condition of containing a predetermined percentage of the predetermined color is hereinafter referred to as the third condition. This configuration facilitates the extraction of a region that is unique within the operating environment as a composite-destination region. As a result, it is possible to provide a model generation image that enables the generation of a more precise 3D model.

[0029] It is preferable that the composite destination extraction unit 13 selects an area of ​​a predetermined shape that satisfies not only the first condition but also the third condition as an area that meets the conditions for the composite destination area. In this way, even if the first condition alone is not enough to narrow down the area to a unique area within the operating environment, adding the third condition also makes it easier to narrow down the area to a unique area. As a result, it is possible to provide an image for model generation that enables the generation of a more precise 3D model. Furthermore, the composite destination extraction unit 13 may select an area of ​​a predetermined shape that satisfies not only the first condition and the second condition but also the third condition as an area that meets the conditions for the composite destination area. This makes it possible to generate a more precise 3D model than when only the first and second conditions are used.

[0030] Here, an example of processing related to the synthesis of images for model generation when CAD data is used in the image acquisition device 1 for 3D model generation (hereinafter referred to as image synthesis-related processing) will be described using the flowchart in Fig. 3. The flowchart in Fig. 3 may be started, for example, when a button operation for starting the image synthesis-related processing is received. Here, the explanation will be given taking as an example a case where the image acquisition device 1 for 3D model generation has already acquired CAD data and marker images.

[0031] First, in step S1, the synthesis-destination extraction unit 13 selects a target region to be extracted as a synthesis region from a CAD model generated from CAD data of the operating environment. This selection can be performed by the synthesis-destination extraction unit 13 searching for a region that meets the conditions of the synthesis-destination region from among the regions of the CAD model. Note that the synthesis-destination extraction unit 13 may be configured to select a target region from among the regions of the CAD model in accordance with a selection input received from a user via an operation input unit.

[0032] In step S2, the captured image acquisition unit 11 acquires the captured image by accepting input of the captured image of the operating environment. In step S3, the synthesis destination extraction unit 13 extracts, from the captured image acquired in S2, an area corresponding to the target area selected in S1 as a synthesis destination area using image recognition technology.

[0033] In step S4, the image synthesis unit 14 synthesizes the marker image with the synthesis destination area extracted in S3 to generate an image for model generation. In step S5, the image output unit 15 outputs the image for model generation generated in S4, and the image synthesis-related processing ends.

[0034] Furthermore, if design data for the operating environment is not available, the composite destination extraction unit 13 may perform the following. The composite destination extraction unit 13 may extract, from the captured image, an area that meets the conditions for the composite destination area, using information specific to an area in the operating environment that meets the conditions for the composite destination area, which information has been pre-learned by machine learning. The information specific to the area that meets the conditions for the composite destination area may be information on a specific object such as a sticker, a teaching pendant, a screw, a button, or an area with a specific shape. The sticker may be, for example, a sticker for use as a warning or instruction manual attached to equipment included in the operating environment. It is preferable that the specific object is one that is likely to be unique in the operating environment. The composite destination extraction unit 13 may extract the position of the specific object, which has been pre-learned by machine learning, from the captured image, using, for example, image recognition technology such as semantic segmentation.

[0035] With the above configuration, even when the design data of the operating environment is unavailable, it is possible to extract a unique area within the operating environment as a synthesis target area. As a result, even when the design data of the operating environment is unavailable, it is possible to provide a model generation image that enables the generation of a more precise 3D model. Note that when extracting multiple regions that are distinguishable from one another, the synthesis target extraction unit 13 may extract the regions of different specific objects as synthesis target areas.

[0036] Here, an example of image synthesis-related processing in the image acquisition device 1 for generating a 3D model when CAD data is not used will be described using the flowchart in Fig. 4. The flowchart in Fig. 4 may also be started, for example, when a button operation for starting the image synthesis-related processing is received. Here, the description will be given taking as an example a case where the image acquisition device 1 for generating a 3D model has already acquired a marker image.

[0037] First, in step S21, the synthesis destination extraction unit 13 selects a target area to be extracted as a synthesis area from the operating environment using a program, etc. For example, the above-mentioned specific area such as an easily recognizable rectangular area or a sticker may be selected as the target area.

[0038] In step S22, the same process as in S2 may be performed. In step S23, the synthesis destination extraction unit 13 extracts, from the captured image acquired in S22, an area corresponding to the target area selected in S21, as a synthesis destination area using image recognition technology.

[0039] In step S24, the image synthesis unit 14 synthesizes the marker image with the synthesis destination area extracted in S23 to generate an image for model generation. In step S25, the image output unit 15 outputs the image for model generation generated in S24, and the image synthesis-related processing ends.

[0040] (Embodiment 2) The configuration is not limited to that of the above-described embodiment, and may be that of the following embodiment 2. An example of the configuration of embodiment 2 will be described below with reference to the drawings.

[0041] <Schematic configuration of image acquisition device 1a for generating 3D model> First, the schematic configuration of image acquisition device 1a for generating a 3D model will be described with reference to Fig. 5. As shown in Fig. 5, image acquisition device 1a for generating a 3D model has, as functional blocks, a captured image acquisition unit 11, a marker image acquisition unit 12a, a compositing destination extraction unit 13a, an image compositing unit 14, an image output unit 15, and an identifier extraction unit 16. Image acquisition device 1a for generating a 3D model has marker image acquisition unit 12a instead of marker image acquisition unit 12. Image acquisition device 1a for generating a 3D model has compositing destination extraction unit 13a instead of compositing destination extraction unit 13. Image acquisition device 1a for generating a 3D model has identifier extraction unit 16. Except for these points, image acquisition device 1a for generating a 3D model is similar to image acquisition device 1 for generating a 3D model of embodiment 1.

[0042] In the second embodiment, a specific identifier is assigned in advance to a location in the operating environment where a reference marker is to be assigned. The specific identifier is an identifier that can read information associated with the marker image. For example, the specific identifier may be a two-dimensional code that includes address information at which the marker image is stored in a storage device. The following description will be given taking as an example a case where the specific identifier is a two-dimensional code. An example of a two-dimensional code is a QR code (registered trademark). The storage device that stores the marker image may be a server external to the image acquisition device 1a for generating a 3D model. Note that the storage device that stores the marker image may be a non-volatile memory internal to the image acquisition device 1a for generating a 3D model.

[0043] As shown in FIG. 6, the storage device that stores the marker images stores different marker images for each address. The marker images stored in the storage device are marker images that can be recognized as reference markers using the current software of the 3D model generation device. FIG. 6 is a diagram for explaining an example of the correspondence between addresses and marker images. Different marker images mean that the 3D model generation device can recognize each marker image as different. Note that the addresses are included in the two-dimensional codes attached to the operating environment, so that a marker image is associated with each different two-dimensional code. In the following, the explanation will be continued using an example in which the storage device that stores the marker images is a server external to the image acquisition device 1a for 3D model generation.

[0044] The identifier extraction unit 16 extracts an identifier from a captured image in which the identifier is captured, from which information associated with the marker image can be read. In this embodiment, a two-dimensional code containing the address information described above is extracted from a captured image in which the two-dimensional code is captured, as illustrated in FIG. 7. FIG. 7 is a diagram showing an example of a captured image of an operating environment to which a two-dimensional code is attached. The TDC in FIG. 7 indicates a two-dimensional code containing address information. The identifier extraction unit 16 extracts the two-dimensional code from a captured image of the operating environment to which the two-dimensional code is attached. The identifier extraction unit 16 may read the address information contained in the extracted two-dimensional code. The address information may be read by a functional block other than the identifier extraction unit 16 in the image acquisition device for 3D model generation 1a.

[0045] The merge destination extraction unit 13a is the same as the merge destination extraction unit 13 of the first embodiment, except for some differences in processing. The differences will be explained below. The merge destination extraction unit 13a extracts, as a merge destination area, an area in which the two-dimensional code extracted by the identifier extraction unit 16 is located.

[0046] The marker image acquiring unit 12a is similar to the marker image acquiring unit 12 of the first embodiment, except for some differences in processing. The following describes these differences. The marker image acquiring unit 12a acquires a marker image linked to information that can be read from the identifier extracted by the identifier extracting unit 16 from a storage device in which the marker image is stored. In the example of this embodiment, the marker image acquiring unit 12a accesses an area of ​​the storage device of the server indicated by an address that can be read from the two-dimensional code extracted by the identifier extracting unit 16. This access may be performed via a network. Then, the marker image acquiring unit 12a acquires the marker image stored in the accessed area via the network.

[0047] The image composition unit 14 of the second embodiment composites the marker image acquired by the marker image acquisition unit 12a into the composition destination region extracted by the composition destination extraction unit 13a from the captured image acquired by the captured image acquisition unit 11. In the example of this embodiment, the marker image acquired by the marker image acquisition unit 12a is composited into the region of the two-dimensional code indicated by TDC in FIG. 7. As a result, an image for model generation such as that shown in FIG. 8 is generated in which the marker image is composited into the region of the two-dimensional code. As explained in the first embodiment, the marker image is composited by performing image conversion in accordance with the inclination of the surface of the composition destination region in the captured image. FIG. 8 is a diagram showing an example of an image for model generation. MIA in FIG. 8 indicates the marker image after image conversion.

[0048] With the above configuration, even if the fiducial markers used to generate the 3D model are changed due to a change in the software that generates the 3D model, this can be easily accommodated by changing the marker images stored in the storage device, thereby eliminating the need to reinstall fiducial markers compatible with the new software in the actual operating environment.

[0049] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also within the technical scope of the present disclosure. The control unit and method described in the present disclosure may be implemented by a special-purpose computer comprising a processor programmed to execute one or more functions embodied in a computer program. Alternatively, the apparatus and method described in the present disclosure may be implemented by a special-purpose hardware logic circuit. Alternatively, the apparatus and method described in the present disclosure may be implemented by one or more special-purpose computers configured by combining a processor executing a computer program with one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory tangible recording medium. [Explanation of symbols]

[0050] 1, 1a Image acquisition device for generating 3D model, 11 Captured image acquisition unit, 12, 12a Marker image acquisition unit, 13, 13a Synthesis destination extraction unit, 14 Image synthesis unit, 15 Image output unit, 16 Identifier extraction unit

Claims

1. 1. An image acquisition device for generating a 3D model, which acquires a captured image of an operating environment of a robot, for use in generating a 3D model of the operating environment, comprising: a captured image acquisition unit (11) that accepts input of the captured image; a marker image acquisition unit (12, 12a) that receives an input of a marker image that is an image of a marker used as a positional reference when generating the 3D model; a synthesis destination extraction unit (13, 13a) that extracts a synthesis destination area that is an area to which the marker image is to be synthesized from the captured image acquired by the captured image acquisition unit; an image synthesis unit (14) that synthesizes the marker image received by the marker image acquisition unit into the synthesis destination area extracted by the synthesis destination extraction unit in the captured image; and an image output unit (15) that outputs a composite image obtained by combining the captured image with the marker image by the image synthesis unit.

2. 2. The image acquisition device for generating a 3D model according to claim 1, the marker image acquisition unit accepts input of a plurality of types of marker images that are distinguishable from one another as the marker images, the composite destination extraction unit extracts a plurality of regions that are distinguishable from one another as the composite destination regions; The image synthesis unit is an image acquisition device for generating a 3D model that synthesizes multiple types of marker images received as input by the marker image acquisition unit into multiple synthesis destination areas extracted by the synthesis destination extraction unit from the captured image.

3. 3. The image acquisition device for generating a 3D model according to claim 1, The synthesis destination extraction unit (13) is an image acquisition device for generating a 3D model that uses design data of the operating environment to extract, from the captured image, an area that meets the conditions of the synthesis destination area as the synthesis destination area.

4. 4. The image acquisition device for generating a 3D model according to claim 3, The composite destination extraction unit is an image acquisition device for generating a 3D model that extracts a composite destination area by determining an area of ​​a predetermined shape with a predetermined aspect ratio as an area that meets the conditions of the composite destination area.

5. 5. The image acquisition device for generating a 3D model according to claim 4, The compositing destination extraction unit extracts the compositing destination area by determining an area of ​​a predetermined shape that includes a predetermined identifier as an area that meets the conditions of the compositing destination area.

6. 5. The image acquisition device for generating a 3D model according to claim 4, The composite destination extraction unit is an image acquisition device for generating a 3D model that extracts a composite destination area by determining an area of ​​a predetermined shape that contains a predetermined color at a predetermined ratio as an area that meets the conditions for the composite destination area.

7. 3. The image acquisition device for generating a 3D model according to claim 1, The synthesis destination extraction unit (13) is an image acquisition device for generating a 3D model that uses information specific to an area in the operating environment that meets the conditions of the synthesis destination area, which information has been machine-learned in advance, to extract an area from the captured image that meets the conditions of the synthesis destination area as the synthesis destination area.

8. 2. The image acquisition device for generating a 3D model according to claim 1, an identifier extraction unit (16) that extracts an identifier from a captured image in which the identifier is captured and from which information associated with the marker image can be read; The synthesis destination extraction unit (13a) extracts an area in which the identifier extracted by the identifier extraction unit is arranged as the synthesis destination area, The marker image acquisition unit (12a) is an image acquisition device for generating a 3D model that accepts input of the marker image by acquiring the marker image linked to information that can be read from the identifier extracted by the identifier extraction unit from a storage device in which the marker image is stored.

Images