DETECTION APPARATUS, DETECTION METHOD, AND PROGRAM

Abstract

To provide an object detection apparatus, a detection method, and a program.SOLUTION: In an operation support system, a detection apparatus (operation room control unit) of a support server includes: a point cloud data acquisition unit which acquires three-dimensional point cloud data of a three-dimensional space where one or more objects are arranged; and an object detection unit which detects at least a part of a boundary of a first object included in the one or more objects, on the basis of the three-dimensional point cloud data of the three-dimensional space acquired by the point cloud data acquisition unit. The three-dimensional point cloud data of the three-dimensional space includes three-dimensional position information which indicates a three-dimensional position of each of points on a surface of the one or more objects and variability information which represents a degree of variability of intensity of light reflected from each of the points. The object detection unit detects at least a part of the boundary of the first object on the basis of the three-dimensional position information and the variability information.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a detection device, a detection method, and a program. [Background technology]

[0002] Patent Document 1 discloses a tracking device that tracks a plurality of objects by utilizing a distance measuring device that captures an image of a target space in which the plurality of objects exist and measures the three-dimensional positions of the plurality of objects. [Prior art document] [Patent documents] [Patent Document 1] JP 2019-205060 A Summary of the Invention [Means for solving the problem]

[0003] In a first aspect of the present invention, a detection device is provided. The detection device includes, for example, a point cloud data acquisition unit that acquires three-dimensional point cloud data of a three-dimensional space in which one or more objects are arranged. The detection device includes, for example, an object detection unit that detects at least a part of a boundary of a first object included in one or more objects based on the three-dimensional point cloud data of the three-dimensional space acquired by the point cloud data acquisition unit. In the detection device, the point cloud data acquisition unit acquires three-dimensional point cloud data of the three-dimensional space from, for example, a measurement device that outputs three-dimensional point cloud data of the three-dimensional space based on reflected light obtained by irradiating light on one or more objects. In the detection device, the three-dimensional point cloud data of the three-dimensional space includes, for example, three-dimensional position information indicating the three-dimensional positions of each of a plurality of points on the surface of one or more objects. In the detection device, the three-dimensional point cloud data of the three-dimensional space includes, for example, variation information indicating the degree of variation in the intensity of reflected light from each of the plurality of points. In the detection device, the object detection unit detects at least a part of the boundary of the first object based on, for example, the three-dimensional position information and the variation information.

[0004] In the above detection device, the object detection unit may extract a point on the surface of the first object from the plurality of points based on a value of an index indicating the degree of variation at each of the plurality of points. In the above detection device, the object detection unit may extract at least one point, the value of which is within a predetermined first range, from the plurality of points. In the above detection device, the object detection unit may compare, for each of the at least one extracted point, a value of an index indicating the degree of variation at each point with a value of an index indicating the degree of variation at each point adjacent to each point. In the above detection device, the object detection unit may distinguish at least two objects included in one or more objects by determining that the two are points on the surface of the same object when the absolute value of the difference between the two is smaller than a predetermined value, and determining that the two are points on the surfaces of different objects when the absolute value of the difference between the two is larger than a predetermined value.

[0005] In the above detection device, the object detection unit may extract points on the surface of the first object from one or more points whose index values ​​indicating the degree of variation are included within a numerical range determined based on the material of the first object. In the above detection device, the three-dimensional point cloud data in the three-dimensional space may include color information indicating the colors of each of the multiple points. In the above detection device, the object detection unit may detect at least a part of the boundary of the first object based on the three-dimensional position information, the variation information, and the color information.

[0006] In the above detection device, the measurement device may include an irradiation unit that irradiates light onto one or more objects arranged in a three-dimensional space. In the above detection device, the measurement device may include a light receiving unit that is arranged near the irradiation unit and receives reflected light from one or more objects. In the above detection device, the detection device or the measurement device may include a depth calculation unit that calculates a depth indicating a distance from the irradiation unit or the light receiving unit for each of a plurality of points on the surface of one or more objects based on the time when the irradiation unit irradiates light and the time when the light receiving unit receives the reflected light. In the above detection device, the detection device or the measurement device may include an index calculation unit that calculates an index indicating the degree of variation in the intensity of reflected light from each of the plurality of points based on the fluctuation of the depth in a unit period having a predetermined length.

[0007] In the above detection device, the detection device or the measurement device may include a first correction unit that corrects an index indicating a degree of variation in the intensity of reflected light from each of the multiple points based on the depth of each of the multiple points calculated by the depth calculation unit.In the above detection device, the detection device or the measurement device may include a second correction unit that corrects an index indicating a degree of variation in the intensity of reflected light from each of the multiple points based on the size of the light receiving element that constitutes the light receiving unit.

[0008] In the above detection device, the detection device or measurement device may include a third correction unit that corrects an index indicating the degree of variation in the intensity of reflected light from each of the multiple points based on the angle between the normal vector of the light-receiving surface of the light-receiving element constituting the light-receiving unit and the normal vector of the measurement target surface on the surface of one or more objects onto which the light is irradiated.In the above detection device, the detection device or measurement device may include a fourth correction unit that corrects an index indicating the degree of variation in the intensity of reflected light from each of the multiple points based on the intensity of reflected light from each of the multiple points received by the light-receiving unit.

[0009] In a second aspect of the present invention, a detection method is provided. The detection method includes, for example, a point cloud data acquisition step of acquiring three-dimensional point cloud data of a three-dimensional space in which one or more objects are arranged. The detection method includes, for example, an object detection step of detecting at least a part of a boundary of a first object included in one or more objects based on the three-dimensional point cloud data of the three-dimensional space acquired in the point cloud data acquisition step. In the detection method, the point cloud data acquisition step includes, for example, a step of acquiring three-dimensional point cloud data of the three-dimensional space from a measurement device that outputs three-dimensional point cloud data of the three-dimensional space based on reflected light obtained by irradiating light on one or more objects. In the detection method, the three-dimensional point cloud data of the three-dimensional space includes, for example, three-dimensional position information indicating the three-dimensional positions of each of a plurality of points on the surface of one or more objects. The three-dimensional point cloud data of the three-dimensional space includes, for example, variation information indicating the degree of variation in the intensity of reflected light from each of the plurality of points. In the detection method, the object detection step includes, for example, a step of detecting at least a part of a boundary of the first object based on the three-dimensional position information and the variation information.

[0010] In a third aspect of the present invention, a program is provided. The program may be a program for causing a computer to function as the detection device according to the first embodiment. The program may be a program for causing a computer to execute the detection method according to the second embodiment. A computer-readable medium storing the program may be provided. The computer-readable medium may be a non-transitory computer-readable medium. The computer-readable medium may be a computer-readable recording medium.

[0011] The above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]

[0012] [Figure 1] 1 illustrates an example of a system configuration of a work support system 100. [Diagram 2] An example of a work object 102 is shown diagrammatically. [Diagram 3] An example of a support image 20 is shown diagrammatically. [Figure 4] 4 shows an example of an operation screen 400 for operating the support image 20. [Diagram 5] 2 shows an example of the internal configuration of the rider 124. [Figure 6] 2 shows an example of the internal configuration of the work chamber control unit 162. [Figure 7] An example of a calibration image 700 is shown diagrammatically. [Figure 8] 13 shows an example of the internal configuration of the support terminal control unit 164. [Figure 9] 13 shows an example of the internal configuration of a point cloud data output unit 540. [Figure 10] 13 shows an overview of information processing in the variation index calculation unit 926. [Figure 11] 13 shows an overview of information processing in the first correction unit 932. [Figure 12] 2 shows an overview of information processing in the second correction unit 934 and the third correction unit 936. [Figure 13] 13 shows an example of an image 1300 generated based on the values ​​of the variability index. [Figure 14] An example of the system configuration of a computer 3000 is shown in schematic form. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The present invention will be described below through the embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. In the drawings, the same reference numbers are used for the same or similar parts, and duplicate explanations may be omitted.

[0014] [Outline of the work support system 100] 1 shows an example of a system configuration of a work support system 100. In this embodiment, the details of the work support system 100 will be described using as an example a case where a worker 32 performs a specific task on a work target 102 inside a workroom 110 and a supporter 34 supports the worker 32 from a remote location. Although the details of the specific task are not limited, examples of the task include document creation, manufacturing of an item, maintenance of an item, and repair of an item.

[0015] Specifically, according to this embodiment, the support person 34 operates the support terminal 140 arranged at a location away from the work room 110 to project the support image 20 for supporting the work on the work target 102 from the projector 126 arranged in the work room 110 toward an appropriate position on the surface of the work target 102 or in its vicinity. This allows the worker 32 to perform the work by referring to (i) the position where the support image 20 is projected, (ii) at least one of the color, shape, and pattern of the support image 20, (iii) information presented by the support image 20, etc.

[0016] For example, when a customer (an example of a worker 32) visiting a store prepares a document such as a contract or application form (an example of a work target 102) with the assistance of a clerk (an example of a supporter 34) working at another store away from the store, the customer may not be able to recognize the location where the seal should be placed. In such a case, when the clerk working at the other store accesses the work support system 100 using a terminal provided at the other store and specifies the location of the seal on the screen of the terminal, an image for highlighting the location of the seal is projected onto the seal location on the customer's document. For example, a red doughnut-shaped figure is projected onto the seal location on the customer's document. After that, the clerk tells the customer, "Please put your seal on the red circle," and the customer can easily recognize the location where the seal should be placed.

[0017] Furthermore, according to this embodiment, the work support system 100 constructs a three-dimensional model of the periphery of the work target 102 in a virtual space to support the supporter 34 in specifying the projection position of the support image 20. This allows the supporter 34 to use the three-dimensional model of the work target 102 to instruct the support image 20 to be projected at any point or area on or near the surface of the work target 102. For example, the supporter 34 can rotate the three-dimensional model of the work target 102 in the virtual space. Therefore, the supporter 34 can easily specify any position on the work target 102 while viewing the work target 102 from directly above.

[0018] Furthermore, when the work support system 100 detects the work object 102 or estimates the type of the work object 102 by image recognition processing, the work support system 100 can execute the above image recognition processing by comparing a teacher image obtained by capturing an image of the work object 102 from a specific direction with an image obtained by observing a three-dimensional model of the work object 102 from a specific direction. Therefore, compared to the case where the above image recognition processing is executed by comparing a two-dimensional image of the work object 102 with a teacher image of the work object 102, the load of the image recognition processing is reduced and the accuracy of the image recognition processing is improved.

[0019] Furthermore, according to this embodiment, the work support system 100 (i) irradiates light onto one or more objects arranged in at least a part of the interior of the workroom 110, and (ii) uses a measuring device that outputs three-dimensional point cloud data of at least a part of the interior of the workroom 110 based on light scattered or reflected back from each of a plurality of points on the surface of the one or more objects (sometimes referred to as reflected light), to construct a three-dimensional model of the periphery of the work target 102 in a virtual space. The work support system 100 calculates the depth of each point based on the light intensity of the reflected light from each of the above-mentioned plurality of points. At this time, the work support system 100 calculates an index (sometimes referred to as a variation index) indicating the degree of variation in the light intensity of the reflected light at each point.

[0020] The inventors have found that the degree of variation in the intensity of reflected light at each point is highly dependent on the material of the object. Although the mechanism is not limited to the following, it is presumed that the degree of variation in the intensity of reflected light is highly dependent on the material of the object because the reflectance or absorptance of light, and the characteristics related to diffuse reflection and / or regular reflection, vary depending on the material of the object.

[0021] According to this embodiment, based on the above knowledge, the work support system 100 uses the above-mentioned variation index when detecting the work object 102 from the three-dimensional point cloud data of at least a part of the interior of the workroom 110. As a result, even if it is difficult to detect the work object 102 using, for example, an RGB image or a distance image, the work support system 100 can relatively easily detect the work object 102 from the three-dimensional point cloud data of at least a part of the interior of the workroom 110.

[0022] For example, when the work room 110 is dark, when strong light is shining around the work object 102, when the periphery of the work object 102 is covered by the shadow of another object, or when the pattern of the work object 102 is similar to the pattern around the work object 102, it is difficult to detect the boundary of the work object 102 from the RGB image of the work object 102 placed on the work table 112. On the other hand, when the materials of the work object 102 and the work table 112 are different, the value of the above-mentioned variation index is different between the points on the surface of the work object 102 and the points on the surface of the work table 112. Therefore, by using the above-mentioned variation index instead of or together with the RGB image, the boundary of the work object 102 can be detected relatively easily even in the above-mentioned cases.

[0023] Similarly, for example, when the thickness of the work object 102 is very small, it is difficult to detect the boundary of the work object 102 from the distance image of the work object 102 placed on the work table 112. On the other hand, by using the above-mentioned variation index instead of or together with the distance image, the boundary of the work object 102 can be detected relatively easily even in the above-mentioned case.

[0024] [Overview of the system configuration of the work support system 100] In this embodiment, the work support system 100 includes a work room 110, a support terminal 140, and a support server 160. In this embodiment, a work table 112 is arranged inside the work room 110, and a work area 114 is set around the work table 112. In this embodiment, a display 122, a rider 124, and a projector 126 are also arranged inside the work room 110.

[0025] In this embodiment, the work object 102 is a tangible object that is the target of work performed by the worker 32. It is preferable that a material having a large diffuse reflectance compared to the regular reflectance is arranged on the surface of the work object 102. Paper, a thin film of metal or metal oxide, a thin film of synthetic polymer or natural polymer, a plant leaf, a fungus, or the like may be arranged on the surface of the work object 102. Examples of fungi include mold and mushrooms. The work object 102 may include at least one of paper, a thin film of metal or metal oxide, a thin film of synthetic polymer or natural polymer, a thin film of resin, a plant, a plant leaf, a food, and a fungus. The work object 102 may be paper, a thin film of metal or metal oxide, a thin film of synthetic polymer or natural polymer, a thin film of resin, or a combination or assembly thereof. The work object 102 may be a plant leaf or a food.

[0026] The thickness of the work target 102 may be less than 3 mm, less than 1 mm, less than 0.5 mm, less than 0.3 mm, or less than 0.1 mm. The work target 102 may be a collection of objects less than 3 mm in thickness, a collection of objects less than 1 mm in thickness, a collection of objects less than 0.5 mm in thickness, a collection of objects less than 0.3 mm in thickness, or a collection of objects less than 0.1 mm in thickness.

[0027] When the work object 102 is paper, a thin film of metal or metal oxide, a thin film of synthetic polymer or natural polymer, or a combination or aggregate thereof, the thickness of the work object 102 is very small. Therefore, for example, it is relatively difficult to detect or recognize the work object 102 based on the depth information shown by the distance image. However, as described later, according to one embodiment of the work support system 100, even if the thickness of the work object 102 is very small, the work object 102 can be detected or recognized by utilizing the difference between the reflection characteristic or diffusion characteristic of light on the surface of the work object 102 and the reflection characteristic or diffusion characteristic of light on the surfaces of objects surrounding the work object 102.

[0028] Furthermore, when the work object 102 is a plant leaf, the surface of the leaf may become discolored or bacteria may grow on the surface of the leaf due to disease. However, for example, the color of the leaf varies greatly depending on the imaging conditions (for example, the brightness at the time of imaging), so it is relatively difficult to estimate the health state of the leaf or distinguish between healthy and diseased leaves based on the RGB image of the leaf. However, according to one embodiment of the work support system 100, it is possible to estimate the health state of the leaf or distinguish between healthy and diseased leaves by utilizing the difference between the light reflection characteristics or diffusion characteristics of the surface of a healthy leaf and the light reflection characteristics or diffusion characteristics of the surface of a diseased leaf.

[0029] Furthermore, when the work object 102 is food, the properties of the food may change due to spoilage or fermentation. However, the color change due to spoilage or fermentation is not necessarily large, and it is relatively difficult to detect the progress of spoilage or fermentation of the food based on an RGB image of the food. However, according to one embodiment of the work support system 100, the progress of spoilage or fermentation of the food can be detected by utilizing the difference between the light reflection characteristics or diffusion characteristics of the surface of a food that is not spoiled or fermented and the light reflection characteristics or diffusion characteristics of the surface of a food that has spoiled or fermented.

[0030] In this embodiment, the support terminal 140 includes a support terminal 142 and a controller 144 that functions as a part of the input device of the support terminal 142. In this embodiment, the support terminal 140 also includes a support terminal 146 and a pointing device 148 that functions as a part of the input device of the support terminal 146.

[0031] In this embodiment, the support server 160 includes a workroom control unit 162 and a support terminal control unit 164. Various devices arranged in the workroom 110, the support terminal 140, and the support server 160 can transmit and receive information to and from each other via the communication network 10.

[0032] [Overview of each part related to the work support system 100] In this embodiment, the communication network 10 may be a wired communication transmission path, a wireless communication transmission path, or a combination of a wireless communication transmission path and a wired communication transmission path. The communication network 10 may include a wireless packet communication network, the Internet, a P2P network, a dedicated line, a VPN, a power line communication line, etc.

[0033] The communication network 10 may include (i) a mobile communication network such as a mobile phone network, or (ii) a wireless communication network such as a wireless MAN (e.g., WiMAX (registered trademark)), a wireless LAN (e.g., WiFi (registered trademark)), Bluetooth (registered trademark), Zigbee (registered trademark), NFC (Near Field Communication), etc. Wireless LAN, Bluetooth (registered trademark), Zigbee (registered trademark), and NFC may be examples of short-range wireless communication.

[0034] In this embodiment, the support image 20 may include a graphic or a character string. In this embodiment, the support image 20 may include a message for the worker 32.

[0035] In this embodiment, the work chamber 110 and the work table 112 are used for the worker 32 to perform work. For example, the work object 102 is placed on the work table 112. The material of the work table 112 is preferably different from the material of the work object 102. In this embodiment, the work area 114 includes at least a portion of the work object 102. The work area 114 may be a space including the entire work object 102 and at least a portion of the work table 112.

[0036] In this embodiment, the display 122 presents information to the worker 32. The display 122 may present information to the worker 32 by image and / or sound. For example, the display 122 outputs a video and / or sound of the supporter 34. This allows the worker 32 to carry out work while talking with the supporter 34, like a video conference or a web conference.

[0037] In this embodiment, the LIDAR 124 acquires three-dimensional point cloud data of the interior of the working room 110. The LIDAR 124 may acquire three-dimensional point cloud data of the interior of the working area 114. For example, the LIDAR 124 irradiates light onto one or more objects disposed within the working area 114 and receives reflected light of the light. The LIDAR 124 outputs three-dimensional point cloud data of one or more objects disposed within the working area 114 based on the reflected light. Details of the LIDAR 124 will be described later.

[0038] In this embodiment, the projector 126 projects the support image 20 onto or near the surface of the work target 102. The projector 126 may project the support image 20 onto or near the surface of the work target 102 in accordance with an instruction from the support server 160. The projector 126 may switch the projection of the support image 20 ON / OFF in accordance with an instruction from the support server 160.

[0039] In this embodiment, the support terminal 140 is used by the supporter 34. The support terminal 140 may function as a user interface when the supporter 34 accesses the work support system 100. For example, the support terminal 140 is used by the supporter 34 to indicate the projection position of the support image 20. Examples of the support terminal 140 include a personal computer and a mobile terminal. Examples of the mobile terminal include a mobile phone, a smartphone, a PDA, a tablet, a notebook computer or laptop computer, and a wearable computer.

[0040] In this embodiment, the support terminal 142 includes a device that provides an XR experience to a user, such as VR (virtual reality) goggles. XR may be a general term for VR (virtual reality), AR (augmented reality), MR (mixed reality), SR (alternative reality), etc. The support person 34 can use the controller 144 to specify the projection position of the support image 20 on the image of the virtual space displayed on the display of the support terminal 142.

[0041] In this embodiment, the support terminal 146 does not include a device that provides an XR experience to the user. Even if the support terminal 146 does not include a device that provides an XR experience to the user, the support terminal 146 can display an image of a virtual space on its display. In addition, the supporter 34 can use the pointing device 148 to specify the projection position of the support image 20 on the image of the virtual space displayed on the display of the support terminal 146.

[0042] In this embodiment, the support server 160 controls the projector 126 based on an instruction from the supporter 34 to project the support image 20 at a position specified by the supporter 34. In addition, the support server 160 recognizes each of one or more objects arranged inside the working area 114 based on the three-dimensional point cloud data of the working area 114 output by the LIDAR 124. For example, the support server 160 detects the work target 102 based on the three-dimensional point cloud data of the working area 114 output by the LIDAR 124.

[0043] In one embodiment, the support server 160 acquires, for example, three-dimensional point cloud data of one or more objects arranged in the working area 114. The support server 160 constructs a virtual space including a three-dimensional model of one or more objects arranged in the working area 114 based on the three-dimensional point cloud data. The support server 160 also receives various instructions from the supporter 34 via the support terminal 140. For example, the support server 160 receives an instruction (sometimes referred to as a first instruction) for identifying a point or area (sometimes referred to as an instruction position) arranged on the surface of the three-dimensional model from the support terminal 140 used by the supporter 34. The support server 160 controls the projector 126 to project the support image 20 onto a point or area (sometimes referred to as a projection position) corresponding to the instruction position within the working area 114.

[0044] In another embodiment, the support server 160 acquires three-dimensional point cloud data of one or more objects arranged in the working area 114, for example, from the lidar 124. The three-dimensional point cloud data of the working area 114 includes three-dimensional position information indicating the three-dimensional positions of each of a plurality of points on the surface of one or more objects arranged inside the working area 114. The three-dimensional point cloud data of the working area 114 also includes variation information indicating the degree of variation in the intensity of reflected light from each of a plurality of points on the surface of one or more objects arranged inside the working area 114. The support server 160 detects the work target 102 based on the three-dimensional point cloud data of the working area 114. For example, the support server 160 detects at least a part of the boundary of the work target 102 based on the three-dimensional position information and the variation information.

[0045] In this embodiment, the work room control unit 162 controls the operation of each device arranged in the work room 110. In addition, the work room control unit 162 acquires various information from each device arranged in the work room 110. Details of the work room control unit 162 will be described later.

[0046] In this embodiment, the support terminal control unit 164 controls the support terminal 140. The support terminal control unit 164 processes a request from the support terminal 140. For example, the support terminal control unit 164 accepts a request from the support terminal 140, and acquires, generates, or extracts information indicated by the request in response to the request from the support terminal 140. The support terminal control unit 164 transmits the above information to the support terminal 140.

[0047] The support terminal control unit 164 also receives requests from the support terminal 140 to obtain information from each device arranged in the work room 110, or requests to control the operation of each device arranged in the work room 110. The support terminal control unit 164 outputs these requests to the work room control unit 162. The support terminal control unit 164 may obtain information indicating the result of the above request from the work room control unit 162, and transmit the information to the support terminal 140. The support terminal control unit 164 will be described in detail later.

[0048] [Specific configuration of each part of the work support system 100] Each part of the work support system 100 may be realized by hardware, software, or hardware and software. At least a part of each part of the work support system 100 may be realized by a single server or by multiple servers. At least a part of each part of the work support system 100 may be realized on a virtual machine or a cloud system. At least a part of each part of the work support system 100 may be realized by a personal computer or a mobile terminal. Examples of the mobile terminal include a mobile phone, a smartphone, a PDA, a tablet, a notebook computer or a laptop computer, and a wearable computer. Each part of the work support system 100 may store information using a distributed ledger technology such as a blockchain or a distributed network.

[0049] When at least some of the components constituting the work assistance system 100 are realized by software, the components realized by the software may be realized by starting software or a program that defines the operation of the components in an information processing device with a general configuration. The information processing device with the general configuration may include (i) a data processing device having a processor such as a CPU or GPU, a ROM, a RAM, a communication interface, etc., (ii) input devices such as a keyboard, a pointing device, a touch panel, a camera, a voice input device, a gesture input device, various sensors, a GPS receiver, etc., (iii) output devices such as a display device, a voice output device, a vibration device, etc., and (iv) storage devices such as a memory, a HDD, a SSD, etc. (including an external storage device).

[0050] In the information processing device of the above general configuration, the above data processing device or storage device may store the above software or program. The above software or program, when executed by a processor, causes the above information processing device to execute the operation defined by the software or program. The above software or program may be stored in a non-transitory computer-readable recording medium. The above software or program may be a program for causing a computer to function as the work support system 100 or a part thereof. The above software or program may be a program for causing a computer to execute an information processing method in the work support system 100 or a part thereof.

[0051] The above information processing method may be an information processing method in the support server 160. The above information processing method may be an information processing method in the work room control unit 162. The above information processing method may be an information processing method in the support terminal control unit 164. The above information processing method may be an information processing method in the rider 124. The above information processing method may be an information processing method in the projector 126.

[0052] In one embodiment, the information processing method includes, for example, a point cloud data acquisition step of acquiring three-dimensional point cloud data of a first object arranged in a three-dimensional space. The information processing method includes, for example, a modeling step of constructing a virtual space including a three-dimensional model of the first object based on the three-dimensional point cloud data acquired in the point cloud data acquisition step. The information processing method includes, for example, an instruction receiving step of receiving a first instruction from a first user for specifying an instruction position that is a point or area arranged on a surface of the three-dimensional model of the first object arranged in the virtual space. The information processing method includes, for example, a projection control step of controlling a projection device that projects an image to project the first image at a projection position that is a point or area corresponding to the instruction position on the surface of the first object.

[0053] In another embodiment, the information processing method includes, for example, a point cloud data acquisition step of acquiring three-dimensional point cloud data of a three-dimensional space in which one or more objects are arranged. The information processing method includes, for example, an object detection step of detecting at least a part of a boundary of a first object included in one or more objects based on the three-dimensional point cloud data of the three-dimensional space acquired in the point cloud data acquisition step. In the information processing method, the point cloud data acquisition step includes, for example, a step of acquiring three-dimensional point cloud data of the three-dimensional space from a measuring device that outputs three-dimensional point cloud data of the three-dimensional space based on reflected light obtained by irradiating light on one or more objects. In the information processing method, the three-dimensional point cloud data of the three-dimensional space includes, for example, three-dimensional position information indicating the three-dimensional positions of each of a plurality of points on the surface of one or more objects. In the information processing method, the three-dimensional point cloud data of the three-dimensional space includes, for example, variation information indicating the degree of variation in the intensity of reflected light from each of the plurality of points. In the information processing method, the object detection step includes, for example, a step of detecting at least a part of a boundary of the first object based on the three-dimensional position information and the variation information.

[0054] The support image 20 may be an example of a first image. The supporter 34 may be an example of a first user. The work target 102 may be an example of a first object or one or more objects. The work target 102 may be an example of a first object included in one or more objects. The work room 110 may be an example of a three-dimensional space. The work table 112 may be an example of a first object or one or more objects. The work area 114 may be an example of a three-dimensional space. One or more objects arranged inside the work area 114 may be an example of a first object. The lidar 124 may be an example of a measuring device. The projector 126 may be an example of a projection device. The controller 144 may be an example of a controller used by the first user. The pointing device 148 may be an example of a controller used by the first user. The support server 160 may be an example of an information processing device or a detection device. The work room control unit 162 may be an example of an information processing device or a detection device.

[0055] [Outline of a method for supporting work using the work support system 100] 2, 3, and 4, an overview of a method for supporting work using the work support system 100 will be described. In this embodiment, an example of a method for supporting work using the work support system 100 will be described, taking as an example a case where the work object 102 is a document such as a contract or an application form. As described above, when the work object 102 is an object with a small thickness such as a document or a film, it is difficult to detect the work object 102 based on a distance image.

[0056] Fig. 2 shows an example of the work object 102. Fig. 3 shows an example of the support image 20. Fig. 4 shows an example of the operation screen 400 for operating the support image 20.

[0057] 2, a title 210 and an encoded code 212 are printed on the surface of the work object 102. The title 210 may be an example of identification information for identifying the type of the work object 102. The encoded code 212 is generated by encoding the identification information for identifying the type of the work object 102. Examples of the encoded code 212 include a barcode and a QR code (registered trademark).

[0058] A writing field 214, a signature field 216, and a seal field 218 are printed on the surface of the work object 102. The writing field 214 is used for the worker 32 to write a character string. The signature field 216 is used for the worker 32 to sign. The seal field 218 is used for the worker 32 to affix a seal. Writing in the writing field 214, signing in the signature field 216, and affixing a seal in the seal field 218 may be examples of work.

[0059] Next, an outline of a procedure for projecting the support image 20 onto the surface of the work target 102 will be described with reference to Figures 3 and 4. As shown in Figure 3, in this embodiment, an arrow-shaped marker 322 is projected at the upper left position of the signature field 216 of the work target 102 so that the supporter 34 can inform the worker 32 of the position of the signature field 216 on the work target 102. Also, a pointer 324 having a frame shape slightly larger than the encoded code 212 is displayed at the position of the encoded code 212 of the work target 102 so that the supporter 34 can inform the worker 32 of the placement position of the work target 102 on the workbench 112.

[0060] In this embodiment, the marker 322 indicates the position of the signature field 216 on the work target 102. Therefore, when the work target 102 moves, it is preferable that the projection position of the marker 322 also moves to follow the movement of the work target 102. Furthermore, when the work target 102 moves in the real space, the change in the real space is also reflected in the virtual space. At this time, the position of the marker 322 in the virtual space may also move to follow the movement of the work target 102 in the virtual space.

[0061] Meanwhile, in this embodiment, the pointer 324 indicates the placement position of the work target 102 on the workbench 112. Therefore, it is preferable that the projection position of the pointer 324 does not change even if the work target 102 moves. The projection position of the pointer 324 in the real space is changed, for example, by the supporter 34 performing an operation to change the position of the pointer 324 in the virtual space.

[0062] Therefore, in this embodiment, when the worker 32 moves the work target 102, the support server 160 controls the projector 126 so that the marker 322 follows the movement of the work target 102. On the other hand, even when the worker 32 moves the work target 102, the support server 160 controls the projector 126 so as to continue projecting the pointer 324 without changing the current projection conditions (for example, at least one of the projection direction of the pointer 324 and the shape and size of the pointer 324). The support server 160 does not need to instruct the projector 126 to change the projection conditions of the pointer 324.

[0063] 4 shows an example of an operation screen 400 for the supporter 34 to input instructions to the support server 160. An example of a mode in which the support server 160 supports the supporter 34 will be described with reference to FIG.

[0064] For example, if the supporter 34 wishes to inform the worker 32 of the position of the signature field 216, the supporter 34 uses the support terminal 140 to access the support server 160. When the supporter 34 accesses the support server 160, an operation screen 400 shown in FIG. 4 is displayed on the display device of the support terminal 140. When the supporter 34 inputs various instructions into the operation screen 400 using the support terminal 140, the support terminal 140 transmits information indicating the content of the instructions of the supporter 34 to, for example, the support terminal control unit 164. The support terminal control unit 164 executes various processes according to the instructions (sometimes referred to as requests) of the supporter 34, and outputs information indicating the execution results to the work room control unit 162 or the support terminal 140.

[0065] According to this embodiment, the supporter 34 uses the controller 144 or the pointing device 148 to specify a point or area (sometimes called a specified position) on the surface of the three-dimensional model of the work target 102 placed in the virtual space on the operation screen 400. As a result, for example, a marker 322 is projected onto a point or area (sometimes called a projection position) on the surface of the work target 102 placed in the real space.

[0066] Specifically, the support person 34 uses the controller 144 or the pointing device 148 to operate a cursor 402 displayed on the operation screen 400 to determine the designated position of the object 404 corresponding to the marker 322. Similarly, the support person 34 can determine various settings related to the object 404. In this embodiment, the cursor 402 indicates a specific point in the virtual space displayed in the virtual space display area 440. In this embodiment, the object 404 corresponds to the support image 20 in the virtual space that is projected into the real space. The shape, color, pattern, and attributes of the object 404 correspond to the shape, color, pattern, and attributes of the support image 20.

[0067] For example, the supporter 34 can use the operation screen 400 to input instructions regarding (i) the position of the support image 20 on the work object 102, (ii) the size of the support image 20 on the work object 102, (iii) at least one of the shape, color, and pattern of the support image 20, and (iv) at least one of the attributes of the support image 20. Examples of the attributes of the support image 20 include (i) whether or not the support image 20 moves following the movement of the work object 102 when the work object 102 moves, and (ii) whether or not the support image 20 is projected onto the surface of the work object 102 while the supporter 34 is operating the support image 20 on the operation screen 400.

[0068] 4, in this embodiment, the operation screen 400 has a menu bar 420, a virtual space display area 440, and an object setting area 460. In this embodiment, an example of the operation screen 400 will be described taking as an example a case where the support image 20 is not projected onto the surface of the work target 102 while the supporter 34 is operating the support image 20 on the operation screen 400.

[0069] In this embodiment, the menu bar 420 displays one or more operation menus. Examples of the operations include operations related to files, operations related to 3D models, operations related to the viewpoint when viewing a 3D model on the operation screen 400, operations related to images projected in the real space (e.g., the marker 322, the pointer 324, etc.), operations related to various settings, etc. Examples of the operations related to the 3D model include operations for reconstructing or updating the 3D model based on the latest point cloud data in the real space.

[0070] In this embodiment, a three-dimensional model of the work area 114 is displayed in the virtual space display area 440. For example, a three-dimensional model 442 of the workbench 112 and a three-dimensional model 444 of the work target 102 are displayed in the virtual space display area 440. Then, for example, the supporter 34 uses the controller 144 or the pointing device 148 to operate the position of the cursor 402 displayed on the operation screen 400, thereby determining the designated position of the object 404 corresponding to the marker 322.

[0071] According to this embodiment, a function for supporting the designation of a position with the cursor 402 in the virtual space display area 440 is provided. For example, in one embodiment, the support terminal control unit 164 supports the movement of the cursor 402 by the supporter 34 in the vicinity of the work target 102. In another embodiment, the support terminal control unit 164 supports the operation of the object 404 by the supporter 34 in the vicinity of the object 404. Details of the function for supporting the designation of a position with the cursor 402 will be described later.

[0072] The object setting area 460 is used to input various settings related to the object 404 to be placed in the virtual space. In this embodiment, the object setting area 460 has a pull-down list 462 for selecting the attributes of the object 404, a pull-down list 464 for selecting the shape, color, and pattern of the object 404, and a button 466 for inputting an instruction to project the support image 20 onto the work target 102 or to reflect changes related to the support image 20.

[0073] According to this embodiment, the supporter 34 can specify the designated position by using the three-dimensional model of the work target 102. Therefore, the process for associating the designated position with the projection position is simplified compared to the case where the designated position is specified on a two-dimensional image. For example, the supporter 34 can rotate the three-dimensional model of the work target 102 on the screen of the support terminal 140, or enlarge a part of the three-dimensional model on the screen of the support terminal 140. The support server 160 may detect an object pointed to by the worker 32 using a part of the body. Examples of the part of the body of the worker 32 include the face, eyes, arms, hands, and fingers. For example, the support server 160 displays an image or an enlarged image of the object in a part of the screen on which the three-dimensional model of the work target 102 is displayed. The support server 160 may display the image or the enlarged image of the object in a pop-up window.

[0074] The instruction regarding the position of the support image 20 on the work target 102 may be an example of a first instruction. The instruction regarding the attribute of the support image 20 may be an example of a second instruction. The cursor 402 may be an example of a mark.

[0075] 5 illustrates an example of the internal configuration of the LIDAR 124. In this embodiment, the LIDAR 124 includes a distance measuring unit 520, an imaging unit 530, and a point cloud data output unit 540. In this embodiment, the distance measuring unit 520 includes an irradiation unit 522 and a light receiving unit 524.

[0076] In this embodiment, the distance measuring unit 520 outputs information for deriving the distance and angle from a reference point of the distance measuring unit 520 for each of a plurality of points on the surface of one or more objects arranged in the working area 114. The distance measuring unit 520 outputs, for example, information indicating the intensity of reflected light from each of a plurality of points on the surface of one or more objects arranged in the working area 114 (sometimes referred to as intensity information). The distance measuring unit 520 may output information for deriving the above-mentioned variation index for each of a plurality of points on the surface of one or more objects arranged in the working area 114.

[0077] In this embodiment, the irradiation unit 522 irradiates light onto one or more objects arranged in the working area 114. Examples of the light include ultraviolet light, visible light, and near-infrared light. The light may be a laser. The light may be pulsed light. The irradiation unit 522 includes, for example, a plurality of light-emitting elements arranged in an array, and an optical system that directs the light emitted by the plurality of light-emitting elements toward the scanning position. Each of the plurality of light-emitting elements is configured to emit light at a predetermined angle with respect to a reference plane of the irradiation unit 522.

[0078] The irradiation unit 522 outputs, for example, information indicating the time when the pulsed light was emitted and the scanning position at that time. The information indicating the scanning position may be an angle between a vector indicating the traveling direction of each light and a normal vector of a reference plane of the irradiation unit 522 that passes through a reference point arranged on the reference plane. The angle between two vectors indicates an angle that is 180° or less among the angles made by the two vectors when at least one of the two vectors is translated to overlap the starting point of one vector with the starting point of the other vector.

[0079] In this embodiment, the light receiving unit 524 is disposed near the irradiation unit 522. The light irradiated by the irradiation unit 522 to the one or more objects is scattered or reflected on the surfaces of the one or more objects and returns to the light receiving unit 524. The light receiving unit 524 receives the light scattered or reflected on the surfaces of the one or more objects and returns (sometimes referred to as reflected light from an object).

[0080] The light receiving unit 524 outputs the fluctuation in the received light intensity of the reflected light. The light receiving unit 524 outputs information indicating, for example, a time and the received light intensity of the reflected light at that time.

[0081] In this embodiment, the imaging section 530 captures an image of the working area 114. The imaging section 530 may capture an image of one or more objects arranged in the working area 114. The imaging section 530 may acquire color information indicating the color of each of a plurality of points on the surface of the one or more objects arranged in the working area 114.

[0082] In this embodiment, the point cloud data output unit 540 generates color point cloud data 560 based on the information output by the distance measurement unit 520 and the information output by the imaging unit 530. As shown in Fig. 5, the color point cloud data 560 stores, for each of a plurality of points on the surface of one or more objects arranged in the working area 114, identification information of each point, three-dimensional position information of each point, color information of each point, and variation information of each point in association with each other. The color point cloud data 560 may further store intensity information indicating the intensity of reflected light from each of a plurality of points on the surface of one or more objects arranged in the working area 114 in association with each other.

[0083] Specifically, for example, the point cloud data output unit 540 calculates a depth indicating a distance from a reference point of the distance measuring unit 520 for each of a plurality of points on the surface of one or more objects arranged in the working area 114 based on the information output by the distance measuring unit 520. The point cloud data output unit 540 may calculate a depth indicating a distance from a reference point of the distance measuring unit 520 for each of the above-mentioned plurality of points based on the time when the distance measuring unit 520 irradiates light and the time when the light receiving unit 524 receives the reflected light of the light. The point cloud data output unit 540 may derive a three-dimensional position of each of the above-mentioned plurality of points based on the above-mentioned depth.

[0084] In addition, the point cloud data output unit 540 calculates a value of a variation index indicating the degree of variation in the intensity of the reflected light from each of the above-mentioned multiple points based on the depth fluctuation in a unit period having a predetermined length. Examples of the above-mentioned index include variance, deviation, standard deviation, etc.

[0085] The point cloud data output unit 540 generates color point cloud data 560 by associating the above-mentioned identification information of each point, three-dimensional position information indicating the three-dimensional position of each point, variation information indicating the value of the variation index of each point, and color information of each point. The point cloud data output unit 540 may output the color point cloud data 560 to the support server 160. This allows the LIDAR 124 to output three-dimensional point cloud data of one or more objects arranged in the working area 114 based on the reflected light received by the light receiving unit 524.

[0086] [An example of another embodiment] In the present embodiment, an example of the work support system 100 has been described, taking as an example a case where variation information is generated inside the LIDAR 124 based on the intensity information output by the light receiving unit 524. However, the work support system 100 is not limited to this embodiment. In other embodiments, the support server 160 may acquire the intensity information output by the light receiving unit 524 and generate variation information based on the intensity information.

[0087] The point cloud data output unit 540 may be an example of a depth calculation unit or an index calculation unit. The color point cloud data 560 may be an example of three-dimensional point cloud data. The distance between each of the multiple points on the surface of one or more objects arranged in the working area 114 and the reference point of the distance measuring unit 520 may be an example of the distance between each of the multiple points on the surface of one or more objects arranged in the working area 114 and the irradiation unit 522 or the light receiving unit 524.

[0088] 6 shows an example of the internal configuration of the working room control unit 162. In this embodiment, the working room control unit 162 includes a point cloud data acquisition unit 622, an object detection unit 632, a modeling unit 634, a type estimation unit 636, an instruction reception unit 642, a projection control unit 644, a calibration unit 650, and a storage unit 660.

[0089] In this embodiment, the point cloud data acquisition unit 622 acquires three-dimensional point cloud data output by the LIDAR 124. For example, the point cloud data acquisition unit 622 acquires color point cloud data 560 related to one or more objects disposed within the working area 114.

[0090] The object detection unit 632 detects at least one of one or more objects (e.g., the work target 102) arranged inside the working area 114 based on the three-dimensional point cloud data acquired by the point cloud data acquisition unit 622. As described above, the three-dimensional point cloud data includes (i) three-dimensional position information or depth information and (ii) at least one of color information, variation information, and intensity information regarding each of a plurality of points on the surface of one or more objects arranged inside the working area 114. In one embodiment, the object detection unit 632 may detect at least one of the one or more objects arranged inside the working area 114 based on the three-dimensional position information or depth information. In another embodiment, the object detection unit 632 may detect at least one of the one or more objects arranged inside the working area 114 based on color information. In yet another embodiment, the object detection unit 632 may detect at least one of the one or more objects arranged inside the working area 114 based on variation information.

[0091] In still another embodiment, the object detection unit 632 may detect at least one of one or more objects arranged inside the working area 114 based on (i) three-dimensional position information or depth information, and (ii) at least one of color information, variation information, and intensity information. For example, the object detection unit 632 first executes an object detection process based on the three-dimensional position information or depth information. When the detection accuracy of at least some objects is smaller than a predetermined value, the object detection unit 632 may detect at least one of the one or more objects arranged inside the working area 114 based on at least one of color information, variation information, and intensity information.

[0092] In still another embodiment, the object detection unit 632 may detect at least one of one or more objects arranged inside the working area 114 based on (i) color information and (ii) at least one of three-dimensional position information or depth information, variation information, and intensity information. For example, the object detection unit 632 first executes an object detection process based on color information. When the detection accuracy of at least some objects is lower than a predetermined value, the object detection unit 632 may detect at least one of one or more objects arranged inside the working area 114 based on at least one of three-dimensional position information or depth information, variation information, and intensity information.

[0093] In still another embodiment, the object detection unit 632 may detect at least one of one or more objects arranged inside the working area 114 based on (i) the variation information and (ii) at least one of the three-dimensional position information or depth information, color information, and intensity information. For example, the object detection unit 632 first executes an object detection process based on the variation information. When the detection accuracy of at least some objects is smaller than a predetermined value, the object detection unit 632 may detect at least one of the one or more objects arranged inside the working area 114 based on at least one of the three-dimensional position information or depth information, color information, and intensity information.

[0094] In this embodiment, when it is known that the surfaces of at least the work target 102 and the work table 112 among one or more objects arranged inside the work area 114 are flat, the object detection unit 632 may detect the work target 102 according to the following procedure. First, the object detection unit 632 uses the three-dimensional point cloud data acquired by the point cloud data acquisition unit 622 before the work target 102 is placed on the work table 112 to approximate the point cloud around the work table 112 with an arbitrary plane equation (sometimes referred to as an approximate plane equation).

[0095] Next, after the work target 102 is placed on the work table 112, the object detection unit 632 reads the three-dimensional point cloud data acquired by the point cloud data acquisition unit 622. Assuming that the work table 112 and the work target 102 are flat, the object detection unit 632 performs projection transformation of the three-dimensional point cloud data near the work table 112 among the three-dimensional point cloud data acquired by the point cloud data acquisition unit 622 into the above-mentioned approximate plane formula. The projection transformation is a coordinate transformation of the point cloud data into a flat and stretched state while maintaining the intermolecular distance of the physically curved, uneven, or folded banknote, as if it has been ironed, even when the work target 102 is a repeatedly curled banknote, etc., and this improves the accuracy of classification or estimation by the object detection unit 632 and / or the type estimation unit 636.

[0096] In this embodiment, the object detection unit 632 detects, for example, the work object 102 based on the three-dimensional point cloud data acquired by the point cloud data acquisition unit 622. The object detection unit 632 may detect the work object 102 by detecting at least a part of the boundary of the work object 102 based on the above-mentioned color point cloud data 560.

[0097] The object detection unit 632 may detect at least a portion of the boundary of the work object 102 based on the three-dimensional position information and variation information included in the color point cloud data 560. The object detection unit 632 may detect at least a portion of the boundary of the work object 102 based on the three-dimensional position information, variation information, and color information included in the color point cloud data 560.

[0098] For example, the object detection unit 632 extracts points on the surface of the work target 102 from among the multiple points based on the value of the variation index for each of the multiple points indicated by the variation information included in the color point cloud data 560. As described above, the value of the variation index differs depending on the material of the object. Therefore, by using the value of the variation index for each point, the object detection unit 632 can distinguish the work target 102 from other objects (e.g., the work table 112) placed in the work area 114 based on the difference in material between the work target 102 and other objects placed in the work area 114. This allows the object detection unit 632 to detect the work target 102 from one or more objects placed in the work area 114.

[0099] More specifically, object detection unit 632 first extracts at least one point whose variation index value is within a predetermined first range from among the above-mentioned multiple points. If the range of possible values ​​of the variation index of work object 102 is known, object detection unit 632 may use the known range as the possible range of values ​​of the variation index of work object 102 as the first range. If the range of possible values ​​of the variation index of a material whose diffuse reflection characteristics are similar to those of work object 102 is known, object detection unit 632 may use the known range as the first range.

[0100] In particular, it is expected that the variation index is proportional to the diffuse reflectance when the diffuse reflectance of the work object 102 to be detected is sufficiently large compared to the specular reflectance of the work object 102. The above first range may be determined taking into consideration the diffuse reflectance of the work object 102 to be detected by the object detection unit 632.

[0101] Next, for each of at least one extracted point, the object detection unit 632 compares the value of the variation index of each point with the value of the variation index of a point adjacent to each point. If the absolute value of the difference between the two is smaller than a predetermined value, the object detection unit 632 determines that the two are points on the surface of the same object. On the other hand, if the absolute value of the difference between the two is larger than a predetermined value, the object detection unit 632 determines that the two are points on the surface of different objects. This allows the object detection unit 632 to distinguish at least two objects included in one or more objects arranged inside the working area 114.

[0102] In addition, if a database that stores the type of material in association with the numerical range that the variation index value can take is available, the object detection unit 632 may simplify the comparison process of the variation index values ​​at the two adjacent points. For example, the object detection unit 632 first uses the material of the work object 102 as a key, refers to the database, and extracts the numerical range that the variation index value of the work object 102 can take. Next, the object detection unit 632 extracts one or more points whose variation index value is included within the extracted numerical range from among a plurality of points on the surface of one or more objects arranged in the work area 114. Next, the object detection unit 632 extracts points on the surface of the work object 102 from the extracted one or more points. The object detection unit 632 may (i) perform a comparison process similar to the above for each of the extracted one or more points, or (ii) detect a closed figure formed by at least a part of the extracted one or more points as the work object 102.

[0103] The object detection unit 632 may detect the contour of each of one or more objects arranged inside the working area 114, and detect the one or more objects. The object detection unit 632 may assign identification information to each of the detected one or more objects. The object detection unit 632 may store the identification information of each object in association with the characteristic information of the appearance of each object. The characteristic information of the appearance of each object is determined, for example, based on an image of each object captured by the imaging unit 530. Thereby, for example, when there are two transfer slips of the same type inside the working area 114, the object detection unit 632 can recognize the two transfer slips as two different instances. For example, when the worker 32 writes characters on one of the transfer slips, the characteristic information of the transfer slip on which the characters are written differs from the characteristic information of the transfer slip on which the characters are written. Therefore, even if the positions of the two transfer slips have moved between the first time, which is the most recent time that the object detection process was performed, and the second time, which is the last time that the object detection process was performed before the first time, the object detection unit 632 can identify the two transfer slips based on the differences in the above-mentioned characteristic information.

[0104] In this embodiment, the modeling unit 634 constructs a virtual space including a three-dimensional model of one or more objects arranged inside the working area 114 based on the three-dimensional point cloud data acquired by the point cloud data acquisition unit 622. The modeling unit 634 may generate a three-dimensional model of each of the objects detected by the object detection unit 632. For example, the modeling unit 634 constructs a virtual space including a three-dimensional model of the work target 102. The modeling unit 634 may construct a virtual space including a three-dimensional model of the work target 102 and a three-dimensional model of the work table 112.

[0105] In this embodiment, the type estimation unit 636 classifies or estimates the type of the object detected by the object detection unit 632. The type estimation unit 636 may classify or estimate the type of the object detected by the object detection unit 632 using a known machine learning method. The type estimation unit 636 may classify or estimate the type of the object using a neural network classifier such as deep learning. The type estimation unit 636 may classify or estimate the type of the object based on an image of the object captured by the imaging unit 530. The type estimation unit 636 may classify or estimate the type of the object based on the boundary of the object detected by the object detection unit 632 and the image of the object captured by the imaging unit 530.

[0106] According to this embodiment, the object detection unit 632 detects the boundaries of one or more objects arranged in the working area 114 based on the value of the variation index. The type estimation unit 636 classifies or estimates the type of each object by using an image of the inside of the boundary of each object, thereby improving the accuracy of the classification or estimation.

[0107] In this embodiment, the instruction receiving unit 642 receives various instructions from the support person 34. The instruction receiving unit 642 may receive instructions inputted on the operation screen 400 by the support person 34 using the support terminal 140.

[0108] The instruction receiving unit 642 may receive an instruction (sometimes referred to as a first instruction) for specifying the above-mentioned indicated position. For example, the supporter 34 can place the object 404 at any position of the three-dimensional model by performing a drag-and-drop operation or a similar operation on the object 404 on the operation screen 400.

[0109] When the supporter 34 completes the operation of placing the object 404, the instruction receiving unit 642 acquires information from the support terminal 140 indicating that the object 404 has been placed at a specific position of the three-dimensional model displayed in the virtual space display area 440 of the operation screen 400. For example, the supporter 34 performs an operation of releasing the button of the pointing device 148 during the drag-and-drop operation of the object 404, thereby completing the operation of placing the object 404. As a result, the instruction receiving unit 642 receives an instruction to specify the position where the object 404 has been placed on the three-dimensional model as the indicated position.

[0110] The instruction receiving unit 642 may receive an instruction (sometimes referred to as a second instruction) for identifying the attribute of the support image 20. For example, the supporter 34 operates the pull-down list 462 for selecting the attribute of the object 404 on the operation screen 400 to select the attribute of the object 404. When the supporter 34 completes the selection operation of the attribute of the object 404, the instruction receiving unit 642 acquires information indicating that the attribute of the object 404 has been determined from the support terminal 140. As a result, the instruction receiving unit 642 receives an instruction for identifying the attribute selected on the operation screen 400 as the attribute of the support image 20.

[0111] The instruction receiving unit 642 receives, for example, (i) an instruction to fix the relative position of the indicated position with respect to a reference point in the three-dimensional model of the work target 102, or (ii) an instruction to fix the relative position of the indicated position with respect to a reference point in the virtual space (sometimes referred to as a third instruction). The third instruction may be (i) a clear instruction indicating that the placement of the object 404 has been completed, or (ii) that the operation for placing the object 404 has been completed. The drag-and-drop operation of the object 404 may be an example of an operation for placing the object 404. In this case, the operation of the supporter 34 releasing the button of the pointing device 148 during the drag-and-drop operation of the object 404 may be an example of the third instruction. An example of a clear instruction indicating that the placement of the object 404 has been completed is pressing the button 466.

[0112] As described above, depending on the attribute of the object 404, the support image 20 is not projected onto the surface of the work target 102 while the supporter 34 is operating the support image 20 on the operation screen 400. In such a case, the third instruction may be an instruction for reflecting a change in the position or setting of the object 404 in the support image 20. Note that whether or not the support image 20 is projected onto the surface of the work target 102 while the supporter 34 is operating the support image 20 on the operation screen 400 may be determined by the setting of the operation screen 400.

[0113] The instruction receiving unit 642 may receive an instruction (sometimes referred to as a fourth instruction) regarding starting or stopping the projection of the support image 20. An example of the fourth instruction is when the button 466 is pressed.

[0114] In this embodiment, the projection control unit 644 controls the operation of the projector 126. The projection control unit 644 controls the projector 126 to project the support image 20, for example, on the surface of the work target 102, onto a point or area (sometimes referred to as a projection position) that corresponds to a designated position on the three-dimensional model of the work target 102.

[0115] The projection control unit 644 may control at least one of the projection direction of the support image 20 and the shape and size of the support image 20. As described above, for example, when the work target 102 moves in real space, the projection control unit 644 controls the projector 126 so that the support image 20 projected onto the work target 102 follows the movement of the work target 102.

[0116] More specifically, when the attribute of support image 20 indicated by the second instruction accepted by instruction accepting unit 642 is the first attribute, projection control unit 644 adjusts at least one of the projection direction of support image 20 in projector 126 and the shape and size of support image 20 so that support image 20 projected onto work object 102 follows the movement of work object 102 when work object 102 moves in real space. Projection control unit 644 may control projector 126 as described above when instruction accepting unit 642 accepts a third instruction and the attribute of support image 20 indicated by the second instruction is the first attribute.

[0117] On the other hand, when the attribute of support image 20 indicated by the second instruction accepted by instruction accepting unit 642 is the first attribute, projection control unit 644 does not adjust the projection direction of support image 20 in projector 126 or the shape and size of support image 20, even if work target 102 moves in real space. When instruction accepting unit 642 accepts a third instruction and the attribute of support image 20 indicated by the second instruction is the second attribute, projection control unit 644 may control projector 126 as described above.

[0118] The projection control unit 644 may control the start or stop of projection of the support image 20. For example, when the instruction receiving unit 642 receives the above-mentioned fourth instruction, the projection control unit 644 may determine whether or not to project the support image 20 onto the work target 102 based on the fourth instruction. The projection control unit 644 may control the operation of the projector 126 based on the above-mentioned determination.

[0119] The projection control unit 644 may control the brightness of the support image 20. In one embodiment, the projection control unit 644 adjusts the brightness of the support image 20 based on the brightness of the surroundings of the projection position in the real space. For example, the projection control unit 644 determines whether the brightness of the surroundings of the projection position is brighter than a predetermined level based on the image data output by the imaging unit 530. If it is determined that the brightness of the surroundings of the projection position is brighter than a predetermined level, the projection control unit 644 reduces the intensity of the projection light of the support image 20. If it is determined that the brightness of the surroundings of the projection position is darker than a predetermined level, the projection control unit 644 increases the intensity of the projection light of the support image 20.

[0120] In another embodiment, the projection control unit 644 controls the brightness of at least a portion of the support image 20 in conjunction with the operation of the type estimation unit 636. The projection control unit 644 may control the brightness of a portion of the support image 20, or may control the brightness of the entire support image 20 in conjunction with the operation of the type estimation unit 636. For example, the projection control unit 644 controls the brightness of a portion of the support image 20 that is projected onto the target of the estimation process by the type estimation unit 636. This suppresses a decrease in estimation accuracy caused by the support image 20 being projected onto the target of the estimation process by the type estimation unit 636.

[0121] For example, when the type estimation unit 636 executes a process of estimating the type of the work object 102 based on the image data output by the imaging unit 530, the projection control unit 644 stops the projection of the support image 20 or reduces the intensity of the projection light of the support image 20. Note that the length of the period during which the projection of the support image 20 is stopped or the period during which the intensity of the projection light of the support image 20 is reduced may be any length that allows the imaging unit 530 to capture an image of the work object 102, and may be, for example, less than one second.

[0122] This allows the type estimation unit 636 to estimate the type of the work object 102 based on image data of the work object 102 captured without the support image 20 being projected. For example, when the work object 102 is a playing card and the shape of the marker 322 or the pointer 324 is heart-shaped, there is a possibility that the estimation accuracy will decrease if the type estimation unit 636 estimates the type of the work object 102 based on image data captured with the marker 322 or the pointer 324 projected onto the work object 102. According to this embodiment, when the type estimation unit 636 executes the above estimation process, for example, the projection of the support image 20 is stopped, so that the estimation accuracy by the type estimation unit 636 is improved.

[0123] When the object of the estimation process by the type estimation unit 636 is the same as the object onto which the support image 20 is projected, the projection control unit 644 may stop the projection of the support image 20 or reduce the intensity of the projection light of at least a part of the support image 20. When the object of the estimation process by the type estimation unit 636 is different from the object onto which the support image 20 is projected, the projection control unit 644 may not stop the projection of the support image 20 or reduce the intensity of the projection light of the support image 20.

[0124] For example, in the case where five objects including object A and object B are arranged inside the working area 114, and part of the support image 20 is projected onto object A but not onto object B, during a period in which the imaging unit 530 is capturing an image of the inside of the working area 114 in order for the type estimation unit 636 to estimate the type of object A, the projection control unit 644 stops the projection of the support image 20 or reduces the intensity of the projection light of at least a part of the support image 20. On the other hand, in the above case, during a period in which the imaging unit 530 is capturing an image of the inside of the working area 114 in order for the type estimation unit 636 to estimate the type of object B, the projection control unit 644 does not need to stop the projection of the support image 20 or reduce the intensity of the projection light of at least a part of the support image 20.

[0125] In this embodiment, the calibration unit 650 performs calibration to determine the positional relationship between the LIDAR 124 and the projector 126. For example, the calibration unit 650 first controls the projector 126 to project a calibration image including at least three line segments whose relative positions are known, onto any plane in the real space. Next, the calibration unit 650 controls the imaging unit 530 of the LIDAR 124 to capture the above-mentioned calibration image. The calibration unit 650 determines the positional relationship between the LIDAR 124 and the projector 126 based on the calibration image captured by the imaging unit 530.

[0126] In this embodiment, the storage unit 660 stores various types of data. The storage unit 660 may store various types of data used for information processing in the support server 160. The storage unit 660 may store various types of data generated by information processing in the support server 160. The storage unit 660 may store three-dimensional model data of one or more objects arranged inside the working area 114. The storage unit 660 may store various types of data used for displaying the operation screen 400. The storage unit 660 may store data related to various settings determined using the operation screen 400.

[0127] [An example of another embodiment] In this embodiment, an example of the workroom control unit 162 has been described, taking as an example a case where the supporter 34 inputs an instruction to the support terminal 140 using the controller 144 or the pointing device 148, and the instruction receiving unit 642 receives information (sometimes referred to as a request) indicating the content of the instruction from the support terminal 140. However, the workroom control unit 162 is not limited to this embodiment. In other embodiments, the instruction of the supporter 34 may be acquired by a device other than the controller 144 or the pointing device 148. For example, a single or multiple cameras arranged on the frame of glasses worn by the supporter 34 detect the shape or movement of the arm, hand, or finger of the supporter 34, and the instruction of the supporter 34 is input to the support terminal 140.

[0128] 7 is a schematic diagram of an example of a calibration image 700. As shown in FIG. 7, the calibration image 700 includes a T-shaped figure. As shown in FIG. 7, the T-shaped figure includes three line segments, AB, BC, and BD, whose relative positions are known. This allows the calibration unit 650 to identify the position of the projector 126 based on (i) the magnitude of an angle AOB between AB and the reference point O of the projector 126, (ii) the magnitude of an angle BOC between BC and the reference point O of the projector 126, and (iii) the magnitude of an angle BOD between BD and the reference point O of the projector 126.

[0129] 8 shows an example of the internal configuration of the support terminal control unit 164. In this embodiment, when the supporter 34 operates a mark (which may be called a pointer, a cursor, or the like) for pointing to a specific point on the screen of the support terminal 140, the support terminal control unit 164 supports the supporter 34 in operating the mark, and an example of the details of the support terminal control unit 164 will be described.

[0130] In this embodiment, the support terminal control unit 164 includes a setting storage unit 810, an indication mark control unit 820, an image data acquisition unit 832, an object estimation unit 834, and a setting extraction unit 836. In this embodiment, the setting storage unit 810 includes a first setting information storage unit 812 and a second setting information storage unit 814.

[0131] In this embodiment, the setting storage unit 810 stores information about various settings used by the information processing in the indication mark control unit 820. The operation of moving the cursor 402 in a three-dimensional virtual space is often more difficult than the operation of moving a mouse cursor on a two-dimensional screen. By applying the above settings, the operation of the cursor 402 in the three-dimensional virtual space can be made easier.

[0132] In this embodiment, the first setting information storage unit 812 stores, for example, first setting information that specifies the manner of movement of the cursor 402 near the boundary of the three-dimensional model of the work target 102. The first setting information storage unit 812 may store the above setting information for each type of work target 102, or may store the above setting information as an initial setting that is applied regardless of the type of the initial work target 102. Examples of the first setting information include (i) a setting regarding the correspondence between the distance from the boundary and the DPI of the controller 144 or the pointing device 148, (ii) a setting regarding whether or not the cursor 402 performs a bouncing action near the boundary, (iii) a setting regarding the bouncing mode when the cursor 402 is bounced, (iv) a setting regarding whether or not the cursor 402 is moved along the boundary for a predetermined period after the cursor 402 reaches the boundary near the boundary, and (v) a setting regarding the moving mode when the cursor 402 is moved along the boundary.

[0133] In this embodiment, the second setting information storage unit 814 stores information indicating the type of each of the one or more objects, and second setting information that specifies the movement of the cursor 402 at one or more positions of the three-dimensional model of each object corresponding to one or more positions on the surface of each object. Taking the work target 102 described in relation to FIG. 2 and FIG. 3 as an example, the one or more positions on the surface of each object are exemplified by the entry field 214, the signature field 216, the seal field 218, etc. The entry field 214, the signature field 216, and the seal field 218 are used relatively frequently. Therefore, if these positions can be easily specified as the above-mentioned specified positions, the burden on the supporter 34 will be greatly reduced. Examples of the second setting information include (i) a setting regarding the correspondence between the above-mentioned positions and the DPI of the controller 144 or the pointing device 148, and (ii) a setting regarding whether or not to fix the cursor 402 at the above-mentioned positions or another predetermined position for a predetermined period when the cursor 402 reaches the above-mentioned positions.

[0134] The indication mark control unit 820 provides a function for supporting the designation of a position by the cursor 402 in the virtual space display area 440. For example, the indication mark control unit 820 controls the movement of the cursor 402 based on a control signal output by the controller 144 or the pointing device 148.

[0135] In one embodiment, the indication mark control unit 820 supports the movement of the cursor 402 by the assistant 34 in the vicinity of the work target 102. The indication mark control unit 820 may switch the control method between a case where the cursor 402 is located in a position near the boundary of the three-dimensional model and a case where the cursor 402 is located in a position other than the vicinity of the boundary of the three-dimensional model. For example, when the cursor 402 is located in a position near the boundary of the three-dimensional model, the indication mark control unit 820 controls the movement of the cursor 402 based on the first setting information. For example, when the position of the object 404 approaches the boundary of the 3D model of the work target 102, the indication mark control unit 820 changes the setting related to the DPI of the controller 144 or the pointing device 148 to reduce the sensitivity of the cursor 402.

[0136] Furthermore, the instruction mark control unit 820 may switch the control method between a case where the cursor 402 is present at a predetermined position on the three-dimensional model and a case where the cursor 402 is present at a position other than the predetermined position on the three-dimensional model. For example, when the cursor 402 is present at a predetermined position on the three-dimensional model, the instruction mark control unit 820 controls the movement of the cursor 402 based on the second setting information extracted by the setting extraction unit 836. For example, when the position of the object 404 approaches a position corresponding to the signature field 216 on the 3D model of the work target 102, the instruction mark control unit 820 moves the cursor 402 to the upper left position of the signature field 216.

[0137] In another embodiment, the instruction mark control unit 820 supports the supporter 34 in operating the object 404 in the vicinity of the object 404. When the cursor 402 approaches the vicinity of an already placed object 404, the instruction mark control unit 820 may change the mode of the workroom control unit 162 to a mode for performing an operation according to the attribute of the object 404. When the cursor 402 moves away from the object 404, the instruction mark control unit 820 may change the mode of the workroom control unit 162 to a mode determined by initial settings.

[0138] For example, if the existing object 404 is a marker 322, when the cursor 402 approaches the object, the instruction mark control unit 820 causes a screen for inputting various settings related to the marker to be displayed in the object setting area 460 of the operation screen 400. On the other hand, if the existing object 404 is a pointer 324, when the cursor 402 approaches the object, the instruction mark control unit 820 causes a screen for inputting various settings related to the pointer to be displayed in the object setting area 460 of the operation screen 400.

[0139] In this embodiment, the image data acquisition unit 832 acquires image data of one or more objects arranged in the working area 114. The image data acquisition unit 832 may acquire color information of the color point cloud data 560 of one or more objects arranged in the working area 114 as image data of the objects. In this embodiment, the object estimation unit 834 estimates the type of each of the one or more objects arranged in the working area 114 based on the image data acquired by the image data acquisition unit 832.

[0140] In this embodiment, the setting extraction unit 836 refers to the second setting information storage unit 814 of the setting storage unit 810, and extracts second setting information corresponding to the type of each object estimated by the object estimation unit 834. As described above, the second setting information is stored in such a manner that information indicating the type of each of the one or more objects is associated with second setting information that specifies the manner of movement of the cursor 402 at each of one or more positions of the three-dimensional model of each object that corresponds to each of one or more positions on the surface of each object.

[0141] 9 illustrates an example of the internal configuration of the point cloud data output unit 540. In this embodiment, the point cloud data output unit 540 includes a depth calculation unit 922, a three-dimensional position information calculation unit 924, a variation index calculation unit 926, and a variation correction unit 930. In this embodiment, the variation correction unit 930 includes a first correction unit 932, a second correction unit 934, a third correction unit 936, and a fourth correction unit 938.

[0142] In this embodiment, the depth calculation unit 922 calculates a depth indicating a distance between each of the plurality of points and a reference point of the distance measurement unit 520 based on information indicating the intensity of reflected light from each of the plurality of points that scatter or reflect the light emitted by the irradiation unit 522. For example, the depth calculation unit 922 calculates the depth of each of the plurality of points based on the time when the distance measurement unit 520 irradiates light and the time when the light receiving unit 524 receives the reflected light of the light. For example, when the distance measurement unit 520 scans the inside of the working area 114, the depth calculation unit 922 calculates the depth for each of the plurality of points on the surface of one or more objects arranged inside the working area 114.

[0143] In this embodiment, the three-dimensional position information calculation unit 924 calculates the three-dimensional position of each of the multiple points based on the depth of each of the multiple points calculated by the depth calculation unit 922. As described above, the irradiation unit 522 outputs information indicating the time when the pulsed light is emitted and the scanning position at that time. The three-dimensional position information calculation unit 924 calculates the distance and angle from the reference point of the distance measurement unit 520 for each of the multiple points based on the information indicating the scanning position and the depth calculated by the depth calculation unit 922.

[0144] In one embodiment, the three-dimensional position information calculation unit 924 outputs, for each of the above-mentioned multiple points, a three-dimensional position based on the reference point of the distance measurement unit 520, based on the distance and angle from the reference point of the distance measurement unit 520. In another embodiment, when the absolute coordinates of the reference point of the distance measurement unit 520 are known, the three-dimensional position information calculation unit 924 may output the absolute coordinates of each of the above-mentioned multiple points, based on the distance and angle from the reference point of the distance measurement unit 520 and the absolute coordinates of the reference point of the distance measurement unit 520.

[0145] In this embodiment, the variation index calculation unit 926 calculates a value of a variation index indicating the degree of variation in the intensity of reflected light from each of the multiple points based on the depth of each of the multiple points calculated by the depth calculation unit 922. The variation index calculation unit 926 may calculate the value of the variation index at each point based on the fluctuation in the depth at each point. For example, the variation index calculation unit 926 calculates the value of the variation index for each of the multiple points based on the fluctuation in the depth in a unit period having a predetermined length. Examples of the above index include variance, deviation, and standard deviation.

[0146] In this embodiment, the variation correction unit 930 corrects the value of the variation index calculated by the variation index calculation unit 926. As described above, the intensity of the reflected light emitted from the distance measurement unit 520 and scattered or reflected on the surface of the work object 102 and returning is significantly affected by the material of the work object 102. This is presumably due to differences in the diffuse reflectance, regular reflectance, and absorptance for light having a specific wavelength, depending on the type of material arranged on the surface of the work object 102, the structure of the surface, and the like.

[0147] In addition to the material of the work target 102, there are other factors (sometimes referred to as error factors) that can cause errors in the value of the variation index. The variation correction unit 930 corrects the value of the variation index in consideration of the above error factors. In one embodiment, the variation correction unit 930 derives a correction coefficient k for correcting the value S of the variation index calculated by the variation index calculation unit 926. The variation correction unit 930 may calculate the above correction coefficient k using a function that has the correction coefficient k as an objective variable and one or more error factors as explanatory variables. The variation correction unit 930 also multiplies the above value S by the correction coefficient k to calculate the corrected value Sd of the variation index. In another embodiment, the variation correction unit 930 derives a function that has the corrected value Sd of the variation index as an objective variable and the above value S and one or more error factors as explanatory variables. The variation correction unit 930 calculates the variation index value Sd after correction based on the function.

[0148] In this embodiment, the first correction unit 932 corrects the value of the variation index calculated by the variation index calculation unit 926, based on the distance between the reference point of the distance measurement unit 520 and the measurement target surface of the surface of the work target 102. The first correction unit 932 may perform a correction process such that the value obtained by subtracting the variation index value Sd after correction from the variation index value S before correction (sometimes referred to as the reduction amount due to correction) becomes larger as the above-mentioned distance increases.

[0149] For example, the first correction unit 932 corrects the value of the variation index of each of the multiple points calculated by the variation index calculation unit 926, based on the depth of each of the multiple points calculated by the depth calculation unit 922. Each of the multiple points may be an example of the measurement target surface.

[0150] As the distance between the reference point of the distance measuring unit 520 and the measurement target surface of the work target 102 increases, the area of ​​the measurement target surface increases. Therefore, the degree of variation in the intensity of reflected light from the measurement target surface may also increase. It can be assumed that the intensity of variation in reflected light from the measurement target surface is proportional to the square of the distance from the surface of a sphere whose center is the point of interest on the reflecting surface and whose radius is the distance to the object. It has been experimentally confirmed that a function in which the intensity of variation in reflected light from the measurement target surface is the objective variable and the distance is the explanatory variable can be approximated by a quadratic equation. Therefore, the first correction unit 932 calculates the corrected variation index value Sd by, for example, dividing the variation index value S calculated by the variation index calculation unit 926 by the square of the distance between the reference point of the distance measuring unit 520 and the measurement target surface of the work target 102.

[0151] This reduces the effect on the value of the variation index of the distance between the distance measuring unit 520 and the work object 102. As a result, for example, when the object detection unit 632 uses the variation index to detect the work object 102 from one or more objects arranged in the work area 114, the difference in the variation index due to the difference between the material of the work object 102 and the materials of the other objects becomes clearer.

[0152] In this embodiment, the second correction unit 934 corrects the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926, based on the size of the light receiving element that constitutes the light receiving unit 524. The second correction unit 934 may perform the correction process such that the amount of reduction due to the correction increases as the size of the light receiving element increases.

[0153] For example, when the shape of the light receiving element can be regarded as a rectangle, the second correction unit 934 corrects the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on the length of the long side of the light receiving element. For example, when the light receiving unit 524 is composed of multiple light receiving elements and at least two of the light receiving elements have different sizes, the second correction unit 934 corrects the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on the size of the light receiving element that outputs the light receiving intensity of the reflected light from each of the multiple points.

[0154] As the size of the light receiving element increases, the area of ​​the measurement target surface increases. Also, as the size of the light receiving element increases, it becomes more susceptible to errors due to the distance between the distance measuring unit 520 and the work target 102. Similarly, as the size of the light receiving element increases, it becomes more susceptible to errors due to the inclination between the reference plane of the distance measuring unit 520 and the measurement target surface of the work target 102. Therefore, the degree of variation in the intensity of the reflected light from the measurement target surface may also increase.

[0155] According to this embodiment, it is possible to reduce the effect of the size of the light receiving element on the value of the variation index by the correction process in the second correction unit 934. As a result, for example, when the object detection unit 632 uses the variation index to detect the work target 102 from one or more objects arranged in the working area 114, the difference in the variation index due to the difference between the material of the work target 102 and the materials of the other objects becomes clearer.

[0156] In this embodiment, the third correction unit 936 corrects the value of the variation index calculated by the variation index calculation unit 926, based on the inclination between the reference plane of the distance measurement unit 520 and the measurement target surface of the work target 102. The third correction unit 936 may perform a correction process such that the value obtained by subtracting the variation index value Sd after correction from the variation index value S before correction (sometimes referred to as the reduction amount due to correction) becomes larger as the reference plane of the distance measurement unit 520 and the measurement target surface of the work target 102 become less parallel.

[0157] For example, the third correction unit 936 corrects the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on the angle between the normal vector of the light receiving surface of the light receiving element that constitutes the light receiving unit 524 and the normal vector of the measurement target surface onto which the light from the irradiation unit 522 is irradiated on the surface of one or more objects to be measured.

[0158] The third correction unit 936 may correct the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on (i) the angle between the normal vector of the light receiving surface of the light receiving element constituting the light receiving unit 524 and the normal vector of the measurement target surface onto which the light from the irradiation unit 522 is irradiated on the surface of one or more objects to be measured, and (ii) the angle between the normal vector of the light receiving surface of the light receiving element constituting the light receiving unit 524 and the vector indicating the traveling direction of the optical axis of the light irradiated onto the measurement target surface.

[0159] The third correction unit 936 may correct the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on (i) the angle between the normal vector of the light receiving surface of the light receiving element constituting the light receiving unit 524 and the normal vector of the measurement target surface on the surface of one or more objects to be measured, onto which the light from the irradiation unit 522 is irradiated, (ii) the angle between the normal vector of the light receiving surface of the light receiving element constituting the light receiving unit 524 and the vector indicating the traveling direction of the optical axis of the light irradiated onto the measurement target surface, and (iii) the size of the light receiving element constituting the light receiving unit 524.

[0160] As the reference surface of distance measuring unit 520 and the measurement surface of work object 102 become less parallel, the difference between the time when reflected light from one end of the measurement surface reaches the light receiving element and the time when reflected light from the other end of the measurement surface reaches the light receiving element increases. Also, as the reference surface of distance measuring unit 520 and the measurement surface of work object 102 become less parallel, the regular reflection component contained in the reflected light decreases, and the received light intensity of the reflected light decreases. Therefore, the degree of inclination between the two can be a factor in errors in the variation index.

[0161] According to this embodiment, it is possible to reduce the effect of the above-mentioned tilt on the value of the variation index by the correction process in the third correction unit 936. As a result, for example, when the object detection unit 632 uses the variation index to detect the work target 102 from one or more objects arranged in the working area 114, the difference in the variation index due to the difference between the material of the work target 102 and the materials of the other objects becomes clearer.

[0162] In this embodiment, the fourth correction unit 938 corrects the value of the variation index calculated by the variation index calculation unit 926 based on the absorptance of the light emitted by the irradiation unit 522 on the measurement target surface. The fourth correction unit 938 may correct the value of the variation index calculated by the variation index calculation unit 926 based on the color of the measurement target surface or the intensity of the reflected light. The fourth correction unit 938 may perform a correction process such that the value obtained by subtracting the variation index value S before correction from the variation index value Sd after correction (which may be referred to as an increase due to correction) becomes larger as the light absorptance increases.

[0163] For example, the fourth correction unit 938 corrects the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on (i) the color of each of the multiple points acquired by the imaging unit 530, or (ii) the intensity of reflected light from each of the multiple points received by the light receiving unit 524. The fourth correction unit 938 may correct the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on the intensity of reflected light from each of the multiple points received by the light receiving unit 524 and the depth of each of the multiple points calculated by the depth calculation unit 922.

[0164] The fourth correction unit 938 may perform the above correction process when the material or color of one or more objects to be detected by the object detection unit 632 is known. The fourth correction unit 938 may perform the above correction process when (i) the size of the working area 114 is known, and (ii) the intensity of the reflected light from each of the multiple points received by the light receiving unit 524 is smaller than a threshold value determined in consideration of the size of the working area 114.

[0165] [An example of another embodiment] In this embodiment, an example of the work support system 100 has been described by taking as an example a case where the point cloud data output unit 540 arranged in the LIDAR 124 calculates the variation index. However, the work support system 100 is not limited to this embodiment. In other embodiments, at least a part of the function of the point cloud data output unit 540 may be realized by the support terminal 140 or the support server 160. For example, at least one of the variation index calculation unit 926 and the variation correction unit 930 is arranged in the support terminal 140 or the support server 160.

[0166] [An example of another embodiment] In the present embodiment, an example of the variation correction unit 930 has been described in which the first correction unit 932, the second correction unit 934, the third correction unit 936, and the fourth correction unit 938 each perform the correction process independently. However, the variation correction unit 930 is not limited to the present embodiment.

[0167] In another embodiment, the variance correction unit 930 may integrally execute the functions of the first correction unit 932, the second correction unit 934, the third correction unit 936, and the fourth correction unit 938. For example, the variance correction unit 930 corrects the value of the variance index calculated by the variance index calculation unit 926 based on at least two combinations of the error factors to be corrected in the first correction unit 932, the error factors to be corrected in the second correction unit 934, the error factors to be corrected in the third correction unit 936, and the error factors to be corrected in the fourth correction unit 938.

[0168] An overview of information processing in the variation index calculation unit 926 will be described with reference to Fig. 10. Fig. 10 shows an example of a depth variation 1020 calculated by the depth calculation unit 922 for a single measurement target surface (e.g., one of the above-mentioned multiple points) on the surface of the work target 102. In Fig. 10, a measurement period Ts indicates a period during which the measurement target surface is irradiated with multiple pulsed lights. An evaluation period Δt indicates an extraction period of samples used by the variation index calculation unit 926 to calculate the variation index.

[0169] The variation index calculation unit 926 calculates a variation index of the depth D by statistically processing the depth D calculated based on the reflected light received by the light receiving unit 524 during the evaluation period Δt. As described above, examples of the variation index include variance, deviation, standard deviation, etc. The variation index calculation unit 926 may calculate an average value Da of the depth D during the evaluation period Δt.

[0170] An overview of information processing in the first correction unit 932 will be described with reference to Fig. 11. Fig. 11 shows a schematic relationship between a distance (e.g., depth D) from a reference point 1124 set on a reference surface 1122 on the light receiving surface of the distance measuring unit 520 and the degree of fluctuation of the depth D during the measurement period Ts described in relation to Fig. 9. As shown in Fig. 11, a comparison of a fluctuation 1142 of the depth D at a depth D1, a fluctuation 1144 of the depth D at a depth D2, and a fluctuation 1146 of the depth D at a depth D3 shows that the fluctuation range increases in this order.

[0171] An overview of information processing in the second correction unit 934 and the third correction unit 936 will be described with reference to Fig. 12. In this embodiment, the distance measuring unit 520 includes a light receiving element 1222 and an optical system 1224. For the purpose of simplifying the description, in this embodiment, the distance measuring unit 520 includes a single light receiving element 1222, and the light receiving element 1222 has a substantially square shape with a side length of c.

[0172] 12, an overview of information processing in the second correction unit 934 and the third correction unit 936 will be described using as an example a case where a normal vector 1226 of the reference surface 1122 passing through the reference point 1124 is not parallel to a vector 1242 indicating the traveling direction of the optical axis of the pulsed light 1240. Also, in FIG. 12, an overview of information processing in the second correction unit 934 and the third correction unit 936 will be described using as an example a case where a normal vector 1226 of the reference surface 1122 of the distance measuring unit 520 is not parallel to a normal vector 1272 of a measurement target surface 1270 arranged on a surface 1260 of the work target 102.

[0173] As the scanning of the surface 1260 progresses, the depth or three-dimensional positions of a plurality of measurement target surfaces 1270 on the surface 1260 are determined. Then, the normal vector 1272 of a particular measurement target surface 1270 can be derived, for example, by calculating the inclination of the surface 1260 near the particular measurement target surface 1270 based on the depth of the particular measurement target surface 1270 and the depths of a plurality of measurement target surfaces 1270 disposed near the particular measurement target surface 1270. Note that in this embodiment, the reference surface 1122 is set on the surface of the optical system 1224. However, the reference surface 1122 may be an example of a light receiving surface of a light receiving element.

[0174] 12, the larger the length c of one side of the light receiving element 1222, the larger the difference in (i) distance between one end 1274 of the single measurement target surface 1270 and the reference point 1124, and the difference in (i) distance between the other end 1276 of the single measurement target surface 1270 and the reference point 1124. It can also be seen that the difference varies depending on the angle between the normal vector 1272 of the measurement target surface 1270 and the vector 1242 indicating the traveling direction of the optical axis of the pulsed light 1240. The angle between normal vector 1272 of measurement surface 1270 and vector 1242 indicating the direction of travel of the optical axis of pulsed light 1240 can be determined, for example, based on (i) the angle between normal vector 1226 of reference surface 1122 and normal vector 1272 of measurement surface 1270, and (ii) the angle between normal vector 1226 of reference surface 1122 and vector 1242 indicating the direction of travel of the optical axis of pulsed light 1240.

[0175] 13 shows an example of an image 1300 generated based on the value of the variation index. The image 1300 is an example of an image that expresses the standard deviation S of the depth D on the measurement target surface in a gray scale of 256 steps. The procedure for creating the image 1300 is as follows.

[0176] First, using Microsoft Azure Kinect DK, we captured images of 1,000 yen, 5,000 yen, and 10,000 yen bills placed on a circular wooden table (Kvistbro, IKEA, blue color) in the dark, and obtained color point cloud data. The table top was painted blue with acrylic paint.

[0177] In addition, when acquiring color point cloud data from Azure Kinect DK, the output of the light intensity of the reflected light from each measurement target surface was extracted from Azure Kinect DK. Based on the fluctuation of the light intensity of the reflected light output from Azure Kinect DK, the standard deviation S of the depth D was calculated for each of the multiple measurement target surfaces. The calculated standard deviation value of each measurement target surface was converted into a 256-level grayscale to create image 1300.

[0178] As described above, the standard deviation S at a specific point indicates the degree of variation in the received light intensity of the reflected light from the specific point. In addition, the standard deviation S at each point was calculated using data from the Azure Kinect DK during a period when the distance value of each point was stable. Specifically, the standard deviation S was calculated according to the following procedure.

[0179] First, the fluctuation of the depth D (i.e., the distance from the Azure Kinect DK) of the specific point or points around the specific point was observed over a predetermined period. Next, data of the depth D in a period in which the fluctuation range was smaller than a predetermined value was extracted. Next, the extracted data of the depth D was statistically processed to calculate the standard deviation S of the depth D. Note that, for a period in which the fluctuation range was larger than a predetermined value, a process of returning a fixed value in the case where the value could not be normally acquired was executed. In this embodiment, 0 was adopted as the fixed value.

[0180] 13, according to the image 1300, the boundaries between the table and each of the 1,000 yen, 5,000 yen, and 10,000 yen bills can be detected by visual inspection. However, since the image was taken in the dark, the boundaries between the table and each of the 1,000 yen, 5,000 yen, and 10,000 yen bills could not be detected based on the RGB image created from the point cloud data. Also, since the 1,000 yen, 5,000 yen, and 10,000 yen bills are very thin, the boundaries between the table and each of the 1,000 yen, 5,000 yen, and 10,000 yen bills could not be detected based on the distance image created from the point cloud data.

[0181] 14 shows an example of a computer 3000 in which multiple aspects of the present invention may be embodied in whole or in part. At least a part of the work support system 100 may be realized by the computer 3000. For example, at least a part of the support server 160 is realized by the computer 3000.

[0182] A program installed on the computer 3000 may cause the computer 3000 to function as or perform operations associated with an apparatus according to an embodiment of the present invention or one or more "parts" of the apparatus, and / or to perform a process or steps of the process according to an embodiment of the present invention. Such a program may be executed by the CPU 3012 to cause the computer 3000 to perform certain operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0183] The computer 3000 according to this embodiment includes a CPU 3012, a RAM 3014, a GPU 3016, and a display device 3018, which are connected to each other by a host controller 3010. The computer 3000 also includes input / output units such as a communication interface 3022, a hard disk drive 3024, a DVD-ROM drive 3026, and an IC card drive, which are connected to the host controller 3010 via an input / output controller 3020. The computer also includes legacy input / output units such as a ROM 3030 and a keyboard 3042, which are connected to the input / output controller 3020 via an input / output chip 3040.

[0184] The CPU 3012 operates according to a program stored in the ROM 3030 and the RAM 3014, thereby controlling each unit. The GPU 3016 acquires image data generated by the CPU 3012 into a frame buffer or the like provided in the RAM 3014 or into the GPU 3016 itself, and causes the image data to be displayed on the display device 3018.

[0185] The communications interface 3022 communicates with other electronic devices via a network. The hard disk drive 3024 stores programs and data used by the CPU 3012 in the computer 3000. The DVD-ROM drive 3026 reads programs or data from the DVD-ROM 3001 and provides the programs or data to the hard disk drive 3024 via the RAM 3014. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0186] The ROM 3030 stores therein a boot program or the like executed by the computer 3000 upon activation, and / or a program that depends on the hardware of the computer 3000. The input / output chip 3040 may also connect various input / output units to the input / output controller 3020 via a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0187] The programs are provided by a computer-readable storage medium such as a DVD-ROM 3001 or an IC card. The programs are read from the computer-readable storage medium, installed in the hard disk drive 3024, the RAM 3014, or the ROM 3030, which are also examples of computer-readable storage media, and executed by the CPU 3012. Information processing described in these programs is read by the computer 3000, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by realizing an operation or processing of information according to the use of the computer 3000.

[0188] For example, when communication is performed between computer 3000 and an external device, CPU 3012 may execute a communication program loaded in RAM 3014 and instruct communication interface 3022 to perform communication processing based on the processing described in the communication program. Under the control of CPU 3012, communication interface 3022 reads transmission data stored in a transmission buffer area provided in RAM 3014, hard disk drive 3024, DVD-ROM 3001, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0189] Furthermore, CPU 3012 may cause all or a necessary portion of a file or database stored in an external recording medium such as hard disk drive 3024, DVD-ROM drive 3026 (DVD-ROM 3001), an IC card, etc. to be read into RAM 3014, and perform various types of processing on the data on RAM 3014. CPU 3012 may then write back the processed data to the external recording medium.

[0190] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 3012 may perform various types of processing on the data read from the RAM 3014, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and writes back the results to the RAM 3014. The CPU 3012 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 3012 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0191] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 3000 or in the vicinity of the computer 3000. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the above-described programs to the computer 3000 via the network.

[0192] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications or improvements can be made to the above embodiments. In addition, the matters described for a specific embodiment can be applied to other embodiments to the extent that they are not technically inconsistent. It is clear from the claims that such modified or improved forms can also be included in the technical scope of the present invention.

[0193] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and may be realized in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is essential to perform the process in this order.

[0194] For example, the present specification discloses the following: [Item A-1] a point cloud data acquisition unit that acquires three-dimensional point cloud data of a first object disposed in a three-dimensional space; a modeling unit that constructs a virtual space including a three-dimensional model of the first object based on the three-dimensional point cloud data acquired by the point cloud data acquisition unit; an instruction receiving unit that receives a first instruction from a first user for identifying a designated position that is a point or an area arranged on a surface of the three-dimensional model of the first object arranged in the virtual space; a projection control unit that controls a projection device that projects an image to project a first image onto a projection position that is a point or area on a surface of the first object corresponding to the designated position; An information processing device comprising: [Item A-2] the projection control unit controls the projection device such that, when the first object moves in the three-dimensional space, the first image projected onto the first object follows the movement of the first object. The information processing device according to item A-1. [Item A-3] the instruction receiving unit receives a second instruction from the first user to identify an attribute of the first image; The projection control unit includes: (i) when the attribute of the first image indicated by the second instruction is a first attribute, adjusting at least one of a projection direction of the first image and a shape and a size of the first image so that the first image projected onto the first object follows the movement of the first object when the first object moves in the three-dimensional space; (ii) when the attribute of the first image indicated by the second instruction is a second attribute, even if the first object moves in the three-dimensional space, the projection direction of the first image and the shape and size of the first image are not adjusted; The information processing device according to item A-1 or A-2. [Item A-4] the instruction receiving unit receives, as an instruction from the first user, (i) a relative position of the designated position with respect to a reference point of the three-dimensional model of the first object, or (ii) a third instruction for fixing a relative position of the designated position with respect to a reference point of the virtual space; The projection control unit includes: (i) when the instruction receiving unit receives the third instruction and the attribute of the first image indicated by the second instruction is a first attribute, when the first object moves in the three-dimensional space, adjusting at least one of a projection direction of the first image and a shape and a size of the first image so that the first image projected onto the first object follows the movement of the first object; (ii) when the instruction receiving unit receives the third instruction and the attribute of the first image indicated by the second instruction is the second attribute, even if the first object moves in the three-dimensional space, the projection direction of the first image and the shape and size of the first image are not adjusted; The information processing device according to item A-3. [Item A-5] the instruction receiving unit receives a fourth instruction from the first user regarding starting or stopping projection of the first image; The projection control unit determines whether or not to project the first image onto the first object based on the fourth instruction. The information processing device according to any one of items A-1 to A-4. [Item A-6] Further, an indication mark control unit controls the movement of a mark indicating a specific point in the virtual space based on a control signal output by a pointing device or a controller used by the first user to specify the indication position in the virtual space. The information processing device according to any one of items A-1 to A-5. [Item A-7] the indication mark control unit controls a movement of the mark based on first setting information that defines a manner of movement of the mark in the vicinity of a boundary of the three-dimensional model of the first object; The information processing device according to item A-6. [Item A-8] an image data acquisition unit that acquires image data of the first object; an object estimation unit that estimates a type of the first object based on the image data acquired by the image data acquisition unit; a setting extraction unit that refers to a setting storage unit that stores information indicating the type of each of the one or more objects in association with second setting information that specifies how the mark moves at each of one or more positions of a three-dimensional model of each object corresponding to each of one or more positions on a surface of each object, and extracts the second setting information corresponding to the type of the first object estimated by the object estimation unit; Further equipped with The indication mark control unit controls the movement of the mark based on the second setting information extracted by the setting extraction unit. The information processing device according to item A-6 or A-7. [Item A-9] the point cloud data acquisition unit acquires the three-dimensional point cloud data of the first object from a measurement device that outputs the three-dimensional point cloud data of the first object based on reflected light of the light obtained by irradiating the first object with light; the measuring device has an imaging unit that images the first object, the information processing device further includes a calibration unit that determines a positional relationship between the measurement device and the projection device; The projection device projects a calibration image including at least three line segments whose relative positions to each other are known into the three-dimensional space; the imaging unit captures the calibration image projected into the three-dimensional space by the projection device; the calibration unit determines a positional relationship between the measurement device and the projection device based on the calibration image captured by the imaging unit. The information processing device according to any one of items A-1 to A-9. [Item A-10] A program for causing a computer to function as the information processing device according to any one of items A-1 to A-10. [Item A-11] A point cloud data acquisition step of acquiring three-dimensional point cloud data of a first object arranged in a three-dimensional space; a modeling step of constructing a virtual space including a three-dimensional model of the first object based on the three-dimensional point cloud data acquired in the point cloud data acquisition step; an instruction receiving step of receiving a first instruction from a first user for identifying a designated position which is a point or an area arranged on a surface of the three-dimensional model of the first object arranged in the virtual space; a projection control step of controlling a projection device that projects an image to project a first image onto a projection position that is a point or area on a surface of the first object corresponding to the indicated position; An information processing method comprising the steps of: [Item B-1] a point cloud data acquisition unit that acquires three-dimensional point cloud data of a three-dimensional space in which one or more objects are arranged; an object detection unit that detects at least a portion of a boundary of a first object included in the one or more objects based on the three-dimensional point cloud data of the three-dimensional space acquired by the point cloud data acquisition unit; Equipped with the point cloud data acquisition unit acquires the three-dimensional point cloud data of the three-dimensional space from a measurement device that outputs the three-dimensional point cloud data of the three-dimensional space based on reflected light of the light obtained by irradiating the one or more objects with light; The three-dimensional point cloud data of the three-dimensional space is Three-dimensional position information indicating a three-dimensional position of each of a plurality of points on a surface of the one or more objects; Variation information indicating a degree of variation in the intensity of the reflected light from each of the plurality of points; and Including, the object detection unit detects at least the part of the boundary of the first object based on the three-dimensional position information and the variation information. Detection device. [Item B-2] The object detection unit is extracting points on the surface of the first object from among the plurality of points based on the value of the index indicating the degree of variation at each of the plurality of points; The detection device according to item B-1. [Item B-3] The object detection unit is extracting at least one point, the value of which of the indexes indicating the degree of variation is within a predetermined first range, from the plurality of points; For each of the at least one extracted point, a value of an index indicating the degree of variation of the point is compared with a value of an index indicating the degree of variation of a point adjacent to the point; When the absolute value of the difference between the two is smaller than a predetermined value, it is determined that the two are points on the surface of the same object, and when the absolute value of the difference between the two is greater than the predetermined value, it is determined that the two are points on the surface of different objects, thereby distinguishing at least two objects included in the one or more objects. The detection device according to item B-2. [Item B-4] the object detection unit extracts a point on the surface of the first object from one or more points having an index value indicating the degree of variation that is included in a numerical range determined based on a material of the first object, among the plurality of points; The detection device according to item B-2 or B-3. [Item B-5] the three-dimensional point cloud data of the three-dimensional space further includes color information indicating the color of each of the plurality of points; the object detection unit detects at least the part of the boundary of the first object based on the three-dimensional position information, the variation information, and the color information. A detection device according to any one of items B-1 to B-4. [Item B-6] The measuring device is an irradiation unit that irradiates the light onto the one or more objects arranged in the three-dimensional space; a light receiving unit disposed near the irradiating unit and configured to receive reflected light of the light from the one or more objects; Equipped with The detection device or the measurement device is a depth calculation unit that calculates a depth indicating a distance from the irradiation unit or the light receiving unit for each of the plurality of points on the surface of the one or more objects based on a time when the irradiation unit irradiates the light and a time when the light receiving unit receives the reflected light; an index calculation unit that calculates an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on a change in the depth in a unit period having a predetermined length; Further comprising: A detection device according to any one of items B-1 to B-5. [Item B-7] The detection device or the measurement device is Further comprising a first correction unit that corrects an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on the depth of each of the plurality of points calculated by the depth calculation unit. A detection device according to item B-6. [Item B-8] The detection device or the measurement device is a second correction unit that corrects an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on a size of a light receiving element that constitutes the light receiving unit; The detection device according to item B-6 or B-7. [Item B-9] The detection device or the measurement device is and a third correction unit that corrects an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on an angle between a normal vector of a light receiving surface of a light receiving element constituting the light receiving unit and a normal vector of a measurement target surface on a surface of the one or more objects onto which the light is irradiated. A detection device according to any one of items B-6 to B-8. [Item B-10] The detection device or the measurement device is Further comprising a fourth correction unit that corrects an index indicating a degree of variation in the intensity of the reflected light from each of the plurality of points based on the intensity of the reflected light from each of the plurality of points received by the light receiving unit. A detection device according to any one of items B-6 to B-9. [Item B-11] A program for causing a computer to function as the detection device according to any one of items B-1 to B-10. [Item B-12] A point cloud data acquisition step of acquiring three-dimensional point cloud data of a three-dimensional space in which one or more objects are arranged; an object detection step of detecting at least a part of a boundary of a first object included in the one or more objects based on the three-dimensional point cloud data of the three-dimensional space acquired in the point cloud data acquisition step; having The point cloud data acquisition step includes a step of acquiring the three-dimensional point cloud data of the three-dimensional space from a measurement device that outputs the three-dimensional point cloud data of the three-dimensional space based on reflected light of the light obtained by irradiating the one or more objects with light, The three-dimensional point cloud data of the three-dimensional space is Three-dimensional position information indicating a three-dimensional position of each of a plurality of points on a surface of the one or more objects; Variation information indicating a degree of variation in the intensity of the reflected light from each of the plurality of points; and Including, The object detection step includes detecting at least a portion of the boundary of the first object based on the three-dimensional position information and the variation information. Detection methods. [Item C] An illumination unit that illuminates one or more objects arranged in a three-dimensional space with light; a light receiving unit disposed near the irradiating unit and configured to receive reflected light of the light from the one or more objects; a depth calculation unit that calculates a depth indicating a distance from the irradiation unit or the light receiving unit for each of the plurality of points on the surface of the one or more objects based on a time when the irradiation unit irradiates the light and a time when the light receiving unit receives the reflected light; an index calculation unit that calculates an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on a change in the depth in a unit period having a predetermined length; A measuring device comprising: [Explanation of symbols]

[0195] 10 communication network, 20 support image, 32 worker, 34 supporter, 100 work support system, 102 work object, 110 work room, 112 work table, 114 work area, 122 display, 124 rider, 126 projector, 140 support terminal, 142 support terminal, 144 controller, 146 support terminal, 148 pointing device, 160 support server, 162 work room control unit, 164 support terminal control unit, 210 title, 212 encoding code, 214 entry field, 216 signature field, 218 seal field, 322 marker, 324 pointer, 400 operation screen, 402 cursor, 404 object, 420 menu bar, 440 virtual space display area, 442 three-dimensional model, 444 three-dimensional model, 460 object setting area, 462 Pull-down list, 464 Pull-down list, 466 Button, 520 Distance measurement unit, 522 Irradiation unit, 524 Light receiving unit, 530 Imaging unit, 540 Point cloud data output unit, 560 Color point cloud data, 622 Point cloud data acquisition unit, 632 Object detection unit, 634 Modeling unit, 636 Type estimation unit, 642 Instruction reception unit, 644 Projection control unit, 650 Calibration unit, 660 Storage unit, 700 Calibration image, 810 Setting storage unit, 812 First setting information storage unit, 814 Second setting information storage unit, 820 Indication mark control unit, 832 Image data acquisition unit, 834 Object estimation unit, 836 Setting extraction unit, 922 Depth calculation unit, 924 Three-dimensional position information calculation unit, 926 Variation index calculation unit, 930 Variation correction unit, 932 First correction unit, 934 Second correction unit, 936 Third correction unit, 938 Fourth correction unit, 1020 Fluctuation, 1122 Reference surface, 1124 Reference point, 1142 Fluctuation, 1144 Fluctuation, 1146 Fluctuation, 1222 Light receiving element, 1224 Optical system, 1226 Normal vector, 1240 Pulsed light, 1242 Vector, 1260 Surface, 1270 Measurement target surface, 1272 Normal vector, 1274 End, 1276 End, 1300 Image, 3000 Computer, 3001 DVD-ROM, 3010 Host controller, 3012 CPU, 3014 RAM, 3016 GPU, 3018 Display device, 3020 Input / output controller, 3022 Communication interface, 3024 Hard disk drive, 3026DVD-ROM drive, 3030 ROM, 3040 I / O chip, 3042 keyboard

Claims

1. A point cloud data acquisition unit that acquires three-dimensional point cloud data of a three-dimensional space in which one or more objects are arranged; an object detection unit that detects at least a portion of a boundary of a first object included in the one or more objects based on the three-dimensional point cloud data of the three-dimensional space acquired by the point cloud data acquisition unit; Equipped with the point cloud data acquisition unit acquires the three-dimensional point cloud data of the three-dimensional space from a measurement device that outputs the three-dimensional point cloud data of the three-dimensional space based on reflected light of the light obtained by irradiating the one or more objects with light; The three-dimensional point cloud data of the three-dimensional space is Three-dimensional position information indicating a three-dimensional position of each of a plurality of points on a surface of the one or more objects; Variation information indicating a degree of variation in the intensity of the reflected light from each of the plurality of points; and Including, The object detection unit is extracting at least one point, the value of which of the indexes indicating the degree of variation at each of the plurality of points is within a predetermined first range, from the plurality of points; For each of the at least one extracted point, a value of an index indicating the degree of variation of the point is compared with a value of an index indicating the degree of variation of a point adjacent to the point; When the absolute value of the difference between the two is smaller than a predetermined value, it is determined that the two are points on the surface of the same object, and when the absolute value of the difference between the two is greater than a predetermined value, it is determined that the two are points on the surface of different objects, thereby distinguishing at least two objects included in the one or more objects. Detection device.

2. the object detection unit extracts a point on the surface of the first object from one or more points, among the plurality of points, where a value of an index indicating the degree of variation is included within a numerical range determined based on a material of the first object; The detection device according to claim 1 .

3. The three-dimensional point cloud data of the three-dimensional space further includes color information indicating a color of each of the plurality of points, the object detection unit detects at least a portion of the boundary of the first object based on the three-dimensional position information, the variation information, and the color information.

3. A detection device according to claim 1 or 2.

4. A detection device, comprising: a point cloud data acquisition unit that acquires three-dimensional point cloud data of a three-dimensional space in which one or more objects are arranged; an object detection unit that detects at least a portion of a boundary of a first object included in the one or more objects based on the three-dimensional point cloud data of the three-dimensional space acquired by the point cloud data acquisition unit; Equipped with the point cloud data acquisition unit acquires the three-dimensional point cloud data of the three-dimensional space from a measurement device that outputs the three-dimensional point cloud data of the three-dimensional space based on reflected light of the light obtained by irradiating the one or more objects with light; The three-dimensional point cloud data of the three-dimensional space is Three-dimensional position information indicating a three-dimensional position of each of a plurality of points on a surface of the one or more objects; Variation information indicating a degree of variation in the intensity of the reflected light from each of the plurality of points; and Including, the object detection unit detects the at least a portion of the boundary of the first object based on the three-dimensional position information and the variation information; The measuring device is an irradiation unit that irradiates the light onto the one or more objects arranged in the three-dimensional space; a light receiving unit disposed near the irradiating unit and configured to receive reflected light of the light from the one or more objects; Equipped with The detection device or the measurement device is a depth calculation unit that calculates a depth indicating a distance from the irradiation unit or the light receiving unit for each of the plurality of points on a surface of the one or more objects based on a time when the irradiation unit irradiates the light and a time when the light receiving unit receives the reflected light; an index calculation unit that calculates an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on a change in the depth in a unit period having a predetermined length; Further comprising: Detection device.

5. The detection device or the measurement device is A first correction unit corrects an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on the depth of each of the plurality of points calculated by the depth calculation unit.

5. The detection device according to claim 4.

6. The detection device or the measurement device is a second correction unit that corrects an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on a size of a light receiving element that constitutes the light receiving unit; 6. A detection device according to claim 4 or claim 5.

7. The detection device or the measurement device is and a third correction unit that corrects an index indicating a degree of variation in the intensity of the reflected light from each of the plurality of points based on an angle between a normal vector of a light receiving surface of a light receiving element constituting the light receiving unit and a normal vector of a measurement target surface on a surface of the one or more objects onto which the light is irradiated.

7. A detection device according to any one of claims 4 to 6.

8. The detection device or the measurement device is and a fourth correction unit that corrects an index indicating a degree of variation in the intensity of the reflected light from each of the plurality of points based on the intensity of the reflected light from each of the plurality of points received by the light receiving unit.

8. A detection device according to any one of claims 4 to 7.

9. A program for causing a computer to function as the detection device according to any one of claims 1 to 8.

10. A point cloud data acquisition step of acquiring three-dimensional point cloud data of a three-dimensional space in which one or more objects are arranged; an object detection step of detecting at least a portion of a boundary of a first object included in the one or more objects based on the three-dimensional point cloud data of the three-dimensional space acquired in the point cloud data acquisition step; having the point cloud data acquisition step includes a step of acquiring the three-dimensional point cloud data of the three-dimensional space from a measurement device that outputs the three-dimensional point cloud data of the three-dimensional space based on reflected light of the light obtained by irradiating the one or more objects with light, The three-dimensional point cloud data of the three-dimensional space is Three-dimensional position information indicating a three-dimensional position of each of a plurality of points on a surface of the one or more objects; Variation information indicating a degree of variation in the intensity of the reflected light from each of the plurality of points; and Including, The object detection step includes: extracting at least one point, the value of which of the indexes indicating the degree of variation at each of the plurality of points is within a predetermined first range, from the plurality of points; For each of the at least one extracted point, a value of an index indicating the degree of variation of the point is compared with a value of an index indicating the degree of variation of a point adjacent to the point; and discriminating between at least two objects included in the one or more objects by determining that the two are points on the surface of the same object when the absolute value of the difference between the two is smaller than a predetermined value, and determining that the two are points on the surface of different objects when the absolute value of the difference between the two is greater than a predetermined value. Detection methods.

11. A detection method comprising: A point cloud data acquisition step of acquiring three-dimensional point cloud data of a three-dimensional space in which one or more objects are arranged; an object detection step of detecting at least a portion of a boundary of a first object included in the one or more objects based on the three-dimensional point cloud data of the three-dimensional space acquired in the point cloud data acquisition step; having the point cloud data acquisition step includes a step of acquiring the three-dimensional point cloud data of the three-dimensional space from a measurement device that outputs the three-dimensional point cloud data of the three-dimensional space based on reflected light of the light obtained by irradiating the one or more objects with light, The three-dimensional point cloud data of the three-dimensional space is Three-dimensional position information indicating a three-dimensional position of each of a plurality of points on a surface of the one or more objects; Variation information indicating a degree of variation in the intensity of the reflected light from each of the plurality of points; and Including, The object detection step includes detecting the at least a portion of the boundary of the first object based on the three-dimensional position information and the variation information, The measuring device is an irradiation unit that irradiates the light onto the one or more objects arranged in the three-dimensional space; a light receiving unit disposed near the irradiating unit and configured to receive reflected light of the light from the one or more objects; Equipped with The detection method includes: a depth calculation step of calculating a depth indicating a distance from the irradiation unit or the light receiving unit for each of the plurality of points on the surface of the one or more objects based on a time when the irradiation unit irradiates the light and a time when the light receiving unit receives the reflected light; an index calculation step of calculating an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on a change in the depth in a unit period having a predetermined length; Further comprising Detection methods.

Images