Placement planning system

Abstract

The present invention makes it possible to easily confirm not only whether or not there is interference between an object in a facility in a real space and equipment to be placed but also how the object and the equipment interfere with each other.　Equipment depth map creation processing 401 creates an equipment depth map M1 for equipment placed according to a placement plan in a coordinate system common to the angle of view of a camera 105. Video depth map creation processing 402 creates a video depth map M2 for a subject in a video captured by the camera 105. Interference detection processing 403 to equipment video appearance modification processing 406 modify an equipment video V2 obtained by drawing the equipment by perspective projection in the coordinate system common to the angle of view of the camera 105, such that a region in which the depth value of the equipment depth map M1 is larger than the depth value of the video depth map M2 and the remaining region have different appearances.

Description

Layout Planning System

[0001] The present invention relates to a layout planning system that supports the creation of layout plans for equipment within a facility.

[0002] Systems that support the creation of layout plans for equipment within facilities such as warehouses and factories are known. For example, Patent Document 1 discloses a system that can create layout plans for equipment and spatial areas in a warehouse by retaining the physical shape of the floor, the shape of logistics equipment such as pallet racks, and the shape of the spatial area in which each item, pallet, container, etc. is placed and stored. The system of Patent Document 1 can also be applied to an assembly factory, where each production cell, etc., is treated as a spatial area and production equipment such as processing machines and racks is treated as logistics equipment, thereby creating layout plans for production equipment and spatial areas.

[0003] Furthermore, Patent Document 2 discloses a surveillance camera system that allows easy confirmation of the correspondence between the actual appearance of the monitored object and the appearance of the three-dimensional shape data by superimposing an image of the monitored object, which is obtained by perspective projection or other methods, onto the captured image of the monitored object.

[0004] JP 2008-094553 A JP 09-009110 A

[0005] OpenCV foundation, “OpenCV”, https: / / docs.opencv.org / 3.4.20 / index.htmlLihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, Hengshuang Zhao, “Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data”, https: / / github.com / LiheYoung / Depth-Anything

[0006] When introducing the equipment layout planning system of Patent Document 1 into an existing facility such as a warehouse or factory to create a layout plan that includes existing equipment, it is necessary to input the shapes and positions of all equipment in the target area within the facility. If this information is not held before introducing the system, it is necessary to survey the equipment existing within the facility and input the shapes and positions, but this becomes a heavy burden when there are a large number of objects to be surveyed.

[0007] Although the system can be used by inputting the shape and position of only a portion of the equipment in the target area within the facility, there is a possibility that a layout plan will be created that interferes with existing equipment that has not been input, resulting in a layout plan that cannot be implemented.One possible solution to this problem is to superimpose the shape of the existing equipment that has been input into the system on the image from a camera installed within the facility, allowing the user to check whether the input equipment is interfering with existing equipment that has not been input.

[0008] However, even if the technology of Patent Document 2 is applied and the shape of the existing equipment input into the system is superimposed on the camera image, the interference between the layout plan and the existing equipment is not displayed separately from other areas of the existing equipment. As a result, there is a possibility that the layout plan, even if it is impossible to implement, may be overlooked. Furthermore, in order for a user to determine whether the layout plan is implementable, it is desirable to be able to easily confirm not only whether or not an object shown in the image within the facility interferes with the equipment in the layout plan, but also how the interference occurs. It is also desirable to be able to confirm whether the interfering object shown in the image within the facility is an existing facility, or a person or vehicle that is temporarily located there.

[0009] The present invention has been made in consideration of the above-described conventional circumstances, and aims to make it possible to easily check not only whether or not an object in a facility in real space interferes with the equipment to be placed, but also how the interference occurs.

[0010] An arrangement planning system according to one embodiment of the present invention includes a first map creation process that creates an equipment depth map in a coordinate system common to the camera's angle of view based on information regarding the three-dimensional shape and arrangement plan of equipment to be arranged within a facility and information regarding the arrangement of a camera that will photograph the facility, the first map creation process creating an equipment depth map that represents depth values ​​relative to the camera for surfaces that are at the back of the camera from among multiple surfaces that represent the three-dimensional shape of the equipment when arranged according to the arrangement plan; a second map creation process that creates a video depth map that represents depth values ​​relative to the camera for subjects in the video photographed by the camera; and a video processing process that processes the equipment video in which the equipment is drawn in a coordinate system common to the camera's angle of view so that areas or their outlines where the depth values ​​of the equipment depth map are greater than the depth values ​​of the video depth map have a different appearance from the remaining areas or their outlines.

[0011] Here, in the above-mentioned layout planning system, the image processing process can be configured to process areas or their contours of the equipment image where the depth value of the equipment depth map is greater than the depth value of the image depth map, or the remaining areas or their contours, so that they have a different appearance by performing a filter process on them.

[0012] In addition, in the above-mentioned layout planning system, the image processing process may be configured to use images previously captured by the camera instead of the equipment image for areas where the depth values ​​of the equipment depth map are greater than the depth values ​​of the image depth map or for remaining areas.

[0013] The above-described layout planning system may further comprise an image superimposition process for superimposing the image processed by the image processing process on the image captured by the camera and outputting the superimposed image.

[0014] According to the present invention, it becomes possible to easily check not only whether or not an object in a facility in real space interferes with the equipment to be placed, but also how the interference occurs.

[0015] 1 is a diagram showing an example of the configuration of a layout planning system according to an embodiment of the present invention; FIG. 2 is a diagram showing an example of classification of information held in the database of FIG. 1; FIG. 3 is a diagram showing an example of functions possessed by the control unit of FIG. 1; FIG. 4 is a diagram showing an example of the flow of processing executed by the layout plan display function of FIG. 3; FIG. 5 is a diagram showing an example of the flow of processing for creating an equipment depth map; FIG. 6 is a diagram showing an example of an image in which an equipment shape is perspectively projected; FIG. 7 is a diagram showing an example of an image in which an equipment depth map is visualized; FIG. 8 is a diagram showing an example of an image captured by a camera; FIG. 9 is a diagram showing an example of an image in which an image depth map is visualized; FIG. 10 is a diagram showing an example of a mask of an interference area; FIG. 11 is a diagram showing an example of a mask of a non-interference area; FIG. 12 is a diagram showing an example of an image in which an equipment image in which the color of the interference area has been changed, an equipment image in the non-interference area, and a camera image are superimposed.

[0016] An embodiment of the present invention will be described with reference to the drawings. Fig. 1 shows an example of the configuration of a layout planning system according to an embodiment of the present invention. The layout planning system shown in the figure includes a database 101, a control unit 102, an operation unit 103, a monitor 104, and a camera 105.

[0017] The database 101 holds (stores) data used to create or display a layout plan. The database 101 reads, creates, modifies, or deletes data in response to a request from the control unit 102. The database 101 may be configured as software that runs on the same computer as the control unit 102, or may be configured as software that runs on a different computer.

[0018] The control unit 102 performs processes related to control of the layout planning system, processes related to functions provided to users, etc. The control unit 102 is realized by, for example, a computer equipped with hardware resources such as a processor and memory, and is configured to realize functions and processes related to the present invention by reading a predetermined program from the memory and executing it with the processor.

[0019] The operation unit 103 receives instructions from the user in accordance with the functions provided to the user by the control unit 102. The operation unit 103 is configured by a keyboard, a mouse, a touch panel, or other devices.

[0020] The monitor 104 displays the video output from the control unit 102. Video signals are transmitted between the control unit 102 and the monitor 104. The monitor 104 has an LCD (Liquid Crystal Display) or a similar display device for displaying the video. It is also possible to connect multiple monitors 104 to the control unit 102. When there are multiple monitors 104 and the control unit 102 outputs multiple videos, each monitor 104 displays the video according to a setting for the combination of the video output by the control unit 102 and the monitor 104 that displays that video.

[0021] Camera 105 is a camera installed within the facility and outputs video captured within the facility to control unit 102. Camera 105 has a CMOS (Complementary Metal-Oxide-Semiconductor) sensor or similar imaging device for capturing video, and an encoder for encoding the video or a processor or electronic circuit for performing similar processing. Video signals are transmitted between camera 105 and control unit 102. Camera 105 may be a camera with a pan / tilt function, and signals indicating pan angles and tilt angles are transmitted between camera 105 and control unit 102. Camera 105 may be a camera with a zoom function, and signals indicating focal length or zoom amount are transmitted between camera 105 and control unit 102. The camera 105 may be a camera equipped with a ToF (Time of Flight) sensor or a similar depth measurement sensor, and depth data and other measurement results are transmitted between the camera 105 and the control unit 102. Multiple cameras 105 can also be connected to the control unit 102.

[0022] Fig. 2 shows an example of classification of information held in the database 101. In the example of Fig. 2, the information held in the database 101 is classified into facility shape information 201, equipment shape information 202, equipment layout plan information 203, spatial range shape information 204, spatial range layout plan information 205, and camera installation information 206.

[0023] Facility shape information 201 is information indicating the three-dimensional shape of the facility's structure (e.g., columns, walls, floors). Equipment shape information 202 is information indicating the three-dimensional shape of equipment (e.g., racks, workbenches, processing machines) that is the subject of the layout plan. Equipment layout plan information 203 is information indicating the position and rotation of each piece of target equipment within the facility, whether the target equipment is to be displayed, etc. Spatial range shape information 204 is information indicating the three-dimensional shape of the spatial range that is the subject of the layout plan (e.g., the range for pallets, the range for containers, the range for cart parking, the range for production cells). The spatial range is a spatial range that represents the storage location and placement destination of indefinite objects, including materials and equipment used for transporting them. Spatial range layout plan information 205 is information indicating the position and rotation of each piece of target spatial range within the facility, whether the target spatial range is to be displayed, etc. Camera installation information 206 is information indicating the installation position, rotation, camera internal parameters, etc. of cameras 105 within the facility.

[0024] The equipment shape information 202 and the equipment layout plan information 203 are both information about equipment that is the subject of a layout plan, and can be collectively treated as equipment-related information 207. The spatial range shape information 204 and the spatial range layout plan information 205 are both information about spatial ranges that are the subject of a layout plan, and can be collectively treated as space range-related information 208. In this example, the spatial range is treated as something separate from equipment, but the spatial range may also be treated as a type of equipment. That is, the "equipment" in the present invention typically refers to equipment related to the equipment-related information 207 (e.g., a rack, a workbench, a processing machine), but may also include space ranges related to the space range-related information 208 (e.g., a range for arranging pallets, a range for arranging containers, a range for parking carts, a range for a production cell).

[0025] Fig. 3 shows an example of functions of the control unit 102. In the example of Fig. 3, the control unit 102 has a placement plan editing function 301, a placement plan display function 302, and an image recording function 303. The control unit 102 may further have a placement plan optimization function 304 for creating optimal placement plan candidates and providing them to the placement plan editing function 301, and other functions for assisting in the creation and implementation of other placement plans.

[0026] The layout plan editing function 301 is a function that accepts requests from the user to create new equipment layout plan information 203 or spatial range layout plan information 205, change existing information, delete existing information, etc. When a user requests the creation of new equipment layout plan information 203 or spatial range layout plan information 205 or the change of existing information, an acceptance condition can be that the equipment or spatial range to be created or changed does not interfere with equipment or spatial ranges related to other equipment layout plan information 203 or spatial range layout plan information 205. Furthermore, when the above request is made, an additional acceptance condition can be that the equipment or spatial range to be created or changed does not interfere with facilities related to the facility shape information 201.

[0027] The layout plan display function 302 is a function for displaying on the monitor 104 the equipment and spatial ranges related to the equipment layout plan information 203 and the spatial range layout plan information 205. The layout plan display function 302 can also use the camera image V1 received from the camera 105 (or one camera 105 selected by the user when multiple cameras 105 are connected) as a background for displaying the equipment and spatial ranges related to the equipment layout plan information 203 and the spatial range layout plan information 205. Past camera image recorded by the image recording function 303 (described later) can also be used as a background for displaying the equipment and spatial ranges related to the equipment layout plan information 203 and the spatial range layout plan information 205. This is effective when analyzing an interference situation in a situation where an object is temporarily placed in a location where the temporarily placed object is not present.

[0028] The image recording function 303 is a function for recording an image of one frame of the camera video V1 received from the camera 105, and can output the image in response to a request from the layout plan display function 302. If there are multiple cameras 105, the image recording function 303 records an image for each camera 105. Image recording can be performed when the system is installed, when requested by the user, when there is no inter-frame difference in the video, or when other conditions for recording are met.

[0029] 4 shows an example of a flow of processing executed by the layout plan display function 302. The following describes processing for displaying equipment related to the equipment layout plan information 203, but it can also be applied to processing for displaying a spatial range related to the spatial range layout plan information 205 or processing for displaying a facility related to the facility shape information 201.

[0030] Here, the facility depth map M1 is defined for all or part of the range of the screen of the camera 105. Each pixel P11 [X11, Y11] on the facility depth map M1 is configured by arranging pixels in N11 rows and N12 columns along the two axes (X-axis and Y-axis) of the screen of the camera 105. Each pixel P11 has a depth value z11 [X11, Y11].

[0031] The facility shape information 203 is assumed to be mesh data that represents the three-dimensional shape of the facility in polygon format. The mesh data has an arbitrary number of vertices v11 [n11]. Each vertex v1 has a position (x21, y21, z21) expressed in an arbitrary coordinate system. The mesh data also has an arbitrary number of faces S1 [n21]. Each face S1 is composed of, for example, three vertices (v12, v13, v14).

[0032] The equipment layout plan information 203 also includes information referencing the corresponding equipment shape information 202, a position t11 at which the equipment is placed relative to a coordinate system defined within the facility, and a rotation R11 that represents the direction and amount of rotation relative to the reference attitude. The position t11 is defined as t11 = (x31, y31, z31), for example. The rotation R11 will be represented below by a rotation matrix. Note that the rotation R11 can also be represented in a format other than a rotation matrix.

[0033] The camera installation information 206 also includes a position t21 and a rotation R21 that position the camera 105 relative to a coordinate system that is common to the position t11 and rotation R11. However, if the camera 105 has a pan / tilt function, the value of the rotation R21 is acquired from the value of a rotation angle sensor of the camera 105. The position t21 is defined as t21 = (x41, y41, z41), for example. The rotation R21 will be represented below by a rotation matrix. Note that the rotation R21 can also be represented in a format other than a rotation matrix.

[0034] In providing the layout plan display function 302, the control unit 102 executes an equipment depth map creation process 401, an image depth map creation process 402, an interference detection process 403, an equipment image perspective projection process 404, an equipment image mask process 405, an equipment image appearance processing process 406, and a camera image superimposition process 407, as shown in FIG. 4 .

[0035] 5 shows an example flow of the facility depth map creation process 401. In the facility depth map creation process 401, first, in process 501, information is checked for each data item in the facility layout plan information 203 to determine whether the target facility is to be displayed. Then, the following processes 502 to 504 are executed for the data item in the facility layout plan information 203 to be displayed.

[0036] In process 502, for each piece of equipment layout plan information 203 data to be displayed (i.e., for each piece of equipment to be placed), the position of each vertex included in the corresponding equipment shape information 202 is converted into a coordinate system based on the optical axis of the camera 105. The position of the vertex before conversion is v21 = (x22, y22, z22), and the position of the vertex after conversion is v31 = (x51, y51, z51). The position of the vertex after conversion can be calculated using the following formula (1).

[0037]

[0038] In process 503, the position v31 of each vertex, transformed into a coordinate system based on the optical axis of the camera 105, is subjected to perspective projection transformation. Specifically, the vertex position v31 is transformed into the position v41 of the vertex, which is represented in a coordinate system consisting of the X-axis and Y-axis on the screen of the camera 105 and the Z-axis based on the camera 105 (the direction behind the camera's viewpoint is positive). The transformed vertex position is set to v41 = (x61, y61, z61). In this transformation, the internal parameters (Fx, Fy, Cx, Cy) of the camera 105 contained in the camera installation information 206 are used as coefficients. Fx and Fy are parameters representing the focal length, and Cx and Cy are parameters representing the optical center (principal point). If the camera 105 has a zoom function, the internal parameters (Fx, Fy, Cx, Cy) of the camera 105 are set to values ​​based on the current zoom state. The position v41 of the vertex after transformation (x61, y61, z61) can be calculated using the following formula (2).

[0039]

[0040] In addition, in order to perform the transformation to obtain position v41 from position v31 with higher accuracy in response to the distortion of the optical system of camera 105, the perspectiveTransform function in the OpenCV library shown in non-patent document 1 or a similar method can be used.

[0041] FIG. 6 shows an example of an equipment video V2 in which the above process 503 is applied to each vertex of the equipment shape information 202 and the surfaces of the equipment shape information 202 are drawn. In FIG. 6, a frame of the equipment video V2 is illustrated as image 601. The screen of the equipment video V2 is configured with the same X-axis and Y-axis as the screen of camera 105. In other words, the equipment video V2 is an image in which the equipment is perspectively projected onto a coordinate system that shares the angle of view of camera 105. In FIG. 6, the result of drawing the equipment shape information 202 is illustrated as equipment image 602 within image 601, which has the same angle of view as camera 105.

[0042] In process 504, the depth value of the facility depth map M1 is updated for each surface of the facility shape information 202. The depth value of the facility depth map M1 is first updated for each of the three vertices that define the surface of the facility shape information 202. If the depth value z12 of pixel P12[X12, Y12] on the facility depth map M1, which is closest to the coordinate position (x61, y61) on the screen of the camera 105 for the vertex position v41=(x61, y61, z61), is not set, then z12[X12, Y12]=z61 is set. If the depth value z12 is set, the value of z12 is set as described above only if z12<z61.

[0043] Next, the depth values ​​of pixels within the range where the surface of the equipment shape information 202 is projected onto the equipment depth map M1 are linearly interpolated based on the depth values ​​of each vertex that defines the surface, or are interpolated by other methods. However, if the depth value z12 is not updated for any vertex of the surface of the equipment shape information 202, the depth value for that surface is not set.

[0044] By performing the above processing, the depth values ​​of the faces in the equipment shape information 202 are reflected in the equipment depth map M1. When multiple faces appear to overlap, the depth values ​​of the faces that are on the far side as seen from the camera 105 are reflected in the equipment depth map M1. By performing the above processing on all data in the equipment layout plan information 203, the depth values ​​of the faces that are on the far side as seen from the camera 105 can be reflected in the equipment depth map M1 for all equipment within the field of view of the camera 105.

[0045] Fig. 7 shows an example of a visualized facility depth map M1 created for the facility shown in image 602. In Fig. 7, a facility depth image 702 that visualizes the facility depth map M1 is drawn within an image 701 that has the same angle of view as the camera 105. The facility depth image 702 represents the depth values ​​set for each pixel on the facility depth map M1 using a color shading pattern, and represents the distribution of depth values ​​for the facility shown as the facility image 602. Note that the shaded areas in image 701 (i.e., areas other than the facility depth image 702) indicate that no depth values ​​have been set.

[0046] Next, the depth map creation process 402 will be described. Here, it is assumed that the depth map M2 is defined for the entire range of the screen of the camera 105. Each pixel P21 [X21, Y21] on the depth map M2 is configured by arranging pixels in N21 rows and N22 columns along the two axes (X-axis and Y-axis) of the screen of the camera 105. Each pixel P21 has a depth value z71 [X21, Y21].

[0047] If the camera 105 is equipped with a sensor (e.g., a Time of Flight sensor) that measures the depth to a subject (such as a facility or existing equipment) for each pixel, the depth values ​​of the depth map M2 can be determined by measurement by the sensor. In this case, the sensor in the camera 105 outputs a depth value z81 for each pixel (X22, Y22) on the screen. The control unit 102 receives the depth value z81 of each pixel from the camera 105. The control unit 102 uses the depth value z81 of each pixel from the camera 105 as the depth value z72 of each pixel P22 [X22, Y22] on the depth map M2 stored in the control unit 102. In other words, z72 [X22, Y22] = z81. By performing this process for all pixels on the screen, the depth map M2 can be created.

[0048] As another method, when a neural network that receives an image and outputs a depth estimation result for each pixel of the image can be used, the depth values ​​of the video depth map M2 can be estimated. In this case, an image captured by the camera 105 is input to the neural network, and the depth values ​​for each pixel output from the neural network are obtained. Here, when a neural network that outputs relative values ​​as the depth values ​​of each pixel is used, conversion to distance is performed. Furthermore, if the units of the depth values ​​output from the neural network are different from the units of the depth values ​​of the facility depth map M1, they are converted to the same units. The control unit 102 sets the value of each pixel as the depth value of each pixel of the video depth map M2. A method of inputting an image into a deep neural network to obtain a depth estimation result is disclosed, for example, in Non-Patent Document 2.

[0049] 8 shows an example of a camera image V1 of the facility captured by the camera 105. In FIG. 8, an image 801 is an example of a frame of the camera image V1.

[0050] 9 shows an example of a depth map M2 created for an image 801 as a depth image 901. The depth image 901 represents the depth values ​​set for each pixel on the depth map M2 as a color shading pattern, and represents the distribution of depth values ​​of subjects (facilities, existing equipment, etc.) shown in the image 801. Note that white areas in the image 801 indicate that depth values ​​have been set but are outside the range of visualization.

[0051] Next, the interference detection process 403 will be described. In the interference detection process 403, interference between equipment and facilities is detected by comparing the depth values ​​of the equipment depth map M1 with the depth values ​​of the video depth map M2. Here, if the number of pixel rows or columns differs between the equipment depth map M1 and the video depth map M2, an enlargement or reduction process is performed on the equipment depth map M1 so that the number of pixel rows or columns of the equipment depth map M1 becomes the same as that of the video depth map M2.

[0052] The interference detection process 403, for example, compares depth values ​​z13 [X31, Y31] and z73 [X31, Y31] at pixels P13 and P23 at the same position (X31, Y31) in the equipment depth map M1 and the image depth map M2, respectively, to create a mask M11 indicating that z13 > z73 and a mask M12 indicating that z13 <= z73. The area indicated by the mask M11 is an area where an object in the camera image V1 and an equipment in the equipment depth map M1 interfere with each other when they are in a common coordinate system (i.e., an interference area). The area indicated by the mask M12 is an area where an object in the camera image V1 and an equipment in the equipment depth map M1 do not interfere with each other when they are in a common coordinate system (i.e., a non-interference area).

[0053] 10 shows an interference region mask image 1001 that visualizes the interference region mask M11 created when the image 601 is the installation image V2 and the image 801 is the camera image V1. The white region in the interference region mask image 1001 is the interference region.

[0054] 11 shows a non-interference region mask image 1101 that visualizes a non-interference region mask M12 that is created when the image 601 is the equipment image V2 and the image 801 is the camera image V1. The white region in the non-interference region mask image 1101 is the non-interference region.

[0055] Next, the equipment image perspective projection process 404 will be described. In the equipment image perspective projection process 404, an image is created in which the equipment is perspectively projected onto a coordinate system that is common to the angle of view of the camera 105. Specifically, the surface S1 of the equipment shape information 202 is drawn on the screen based on the vertex position v41 obtained in process 503 of the equipment depth map creation process 401, to create the equipment image V2. The equipment image V2 can also be created by performing computer graphics processing such as shading. The image 601 shown in FIG. 6 is an example of the equipment image V2.

[0056] Next, the equipment image masking process 405 will be described. In the equipment image masking process 405, a mask M11 is applied to the equipment image V2 to create an interference area image V3. Furthermore, a mask M12 is applied to the equipment image V2 to create a non-interference area image V4. This divides the equipment image V2 into an interference area image V3 and a non-interference area image V4. Note that when a previously captured image F1 recorded in the image recording function 303 is to be displayed in the interference area, the mask M11 can be applied to the captured image F1 to create the interference area image V3. Furthermore, when the image F1 is to be displayed in the non-interference area, the mask M12 can be applied to the captured image F1 to create the non-interference area image V4.

[0057] Next, the equipment image appearance processing 406 will be described. In the equipment image appearance processing 406, the interference area image V3 or the non-interference area image V4 is processed so that the interference area between the object in the camera image V1 and the object in the equipment depth map M1 has a different appearance from the remaining area. The equipment image appearance processing 406 can be, for example, a process of multiplying the pixel values ​​of the interference area image V3 or the non-interference area image V4 by a fixed ratio, a process of applying a specific transparency value to the pixel values ​​of the interference area image V3 or the non-interference area image V4, or a process of applying a tone curve related to a specific brightness to the pixel values ​​of the interference area image V3 or the non-interference area image V4, or the like, but other filter processes may also be used.

[0058] As an example, a process for multiplying pixel values ​​of a specific color and displaying the interference area between equipment in a camera image and equipment in the equipment layout plan information 203 will be described. Each pixel P31 (X41, Y41) in the interference area image V3 has a color pixel value. The color pixel values ​​are expressed as (r1, g1, b1, a1) in the RGBA format. r1, g1, and b1 are values ​​indicating the intensity of red, green, and blue, respectively, and a1 is a value indicating transparency. The color can be changed by changing the values ​​of r1, g1, and b1 for each pixel. For example, the following equation (3) shows a method for calculating the pixel value (r3, g3, b3, a1) of the color of pixel P31 by multiplying the average value of the pixel value of the color of pixel P31 by the value of a predetermined color (r2, g2, b2). Here, mean(·) in the equation is a function for calculating the average value.

[0059]

[0060] Next, the camera image superimposition process 407 will be described. In the camera image superimposition process 407, the interference area image V3, the non-interference area image V4, and the camera image V1 are superimposed in this order from the foreground, to create a composite image V5 in which the equipment image is superimposed on the camera image. The control unit 102 outputs the composite image V5 to the monitor 104.

[0061] 12 illustrates an example of a composite image V5 that is created when an interference area image V3 and a non-interference area image V4 are created by changing the color pixel values ​​based on the equipment image V2 shown in image 601 using the method illustrated in the explanation of the equipment image appearance processing 406, and then superimposing these on the camera image V1 shown in image 801. In Fig. 12, a frame of the composite image V5 is shown as image 1201. A colored area 1202 in image 1201 is the interference area (interference area image V3 after processing), and the color of the equipment image V2 has been changed so that it can be distinguished from the non-interference area V4.

[0062] As described above, in the layout planning system of this example, the facility depth map creation process 401 creates a facility depth map M1 that represents depth values ​​relative to the camera 105 for surfaces that are farther from the camera 105 than those of multiple surfaces that represent the three-dimensional shape of the facility when it is placed according to the layout plan, in a coordinate system common to the angle of view of the camera 105, based on facility shape information 202 related to the three-dimensional shape of the facility to be placed within the facility, facility layout plan information 203 related to the layout plan, and camera installation information 206 related to the placement of the camera 105 that captures the facility. Next, the image depth map creation process 402 creates a image depth map M2 that represents depth values ​​relative to the camera 105 for objects (such as facilities or existing facilities) in the image captured by the camera 105. Next, the interference detection process 403 compares the depth values ​​of the facility depth map M1 with the depth values ​​of the image depth map M2 to detect interference between the facility and the facility. Next, an equipment image perspective projection process 404 creates an equipment image V2 in which the equipment is drawn by perspective projection in a coordinate system that is common to the angle of view of the camera 105. Next, an equipment image mask process 405 and an equipment image appearance processing process 406 process the equipment image V2 so that the area where the depth value of the equipment depth map M1 is greater than the depth value of the image depth map M2 has a different appearance from the remaining area. By performing such processing, an interference area image V3 and a non-interference area image V4 are obtained, so that the user can easily check not only whether or not there is interference between an object in the facility in real space and the equipment to be placed, but also how the interference occurs, simply by viewing these images.

[0063] Furthermore, in the layout planning system of this example, a composite image V5 is created by superimposing the interference area image V3, the non-interference area image V4, and the camera image V1 using a camera image superimposition process 407. Therefore, the user can check how the equipment will look when placed within the facility simply by looking at the composite image V5.

[0064] It is also possible to create an image in which the outline of the interfering area has a different appearance from the outline of the remaining area, rather than an image in which the interference area between the object in the camera image V1 and the object in the equipment depth map M1 has a different appearance from the outline of the remaining area. In this case, a mask M13 for the outline part of mask M11 and a mask M14 that is the inverse of mask M13 are created, and mask M13 is applied instead of mask M11, and mask M14 is applied instead of mask M12.

[0065] Furthermore, instead of the image obtained by the above processing, an image previously captured by the camera 105 may be applied to the interference area image V3 or the non-interference area image V4 of the equipment image V2. In this way, when interference is detected in a location where an object is temporarily placed, it is possible to grasp the situation without the temporarily placed object by using an image previously captured in a situation where the temporarily placed object is not present.

[0066] Although the embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take on various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and modifications thereof are included in the scope and spirit of the invention described in this specification, etc., and are included in the invention described in the claims and their equivalents.

[0067] Furthermore, the present invention can be provided not only as devices such as those described above or as systems composed of these devices, but also as methods executed by these devices, programs for realizing the functions of these devices using a processor, and storage media for storing such programs in a computer-readable manner.

[0068] The present invention can be used in a layout planning system that assists in creating a layout plan for equipment within a facility.

[0069] 101: Database, 102: Control unit, 103: Operation unit, 104: Monitor, 105: Camera, 201: Facility shape information, 202: Equipment shape information, 203: Equipment layout plan information, 204: Spatial range shape information, 205: Spatial range layout plan information, 206: Camera installation information, 207: Equipment related information, 208: Spatial range related information, 301: Layout plan editing function, 302: Layout plan display function, 303: Image recording function, 304: Layout plan optimization function

Claims

1. A layout planning system characterized by having: a first map creation process that creates, in a coordinate system common to the camera's angle of view, an equipment depth map that represents depth values ​​relative to the camera for surfaces that are at the back of the camera from among multiple surfaces that represent the three-dimensional shape of the equipment when placed according to the layout plan, based on information regarding the three-dimensional shape and layout plan of the equipment to be placed within the facility and information regarding the placement of a camera that will photograph the facility; a second map creation process that creates an image depth map that represents depth values ​​relative to the camera for subjects in the image photographed by the camera; and an image processing process that processes the equipment image in which the equipment is drawn in a coordinate system common to the camera's angle of view so that areas or their outlines where the depth values ​​of the equipment depth map are greater than the depth values ​​of the image depth map have a different appearance from the remaining areas or their outlines.

2. In the layout planning system described in claim 1, the image processing is characterized in that it processes areas or their outlines of the equipment image where the depth value of the equipment depth map is greater than the depth value of the image depth map, or the remaining areas or their outlines, so that they have a different appearance.

3. In the layout planning system described in claim 1, the image processing process uses images previously captured by the camera instead of the equipment image for areas where the depth values ​​of the equipment depth map are greater than the depth values ​​of the image depth map or for remaining areas.

4. A layout planning system according to any one of claims 1 to 3, further comprising an image superimposition process for superimposing the image processed by the image processing process onto the image captured by the camera and outputting the superimposed image.

Images