Electronic device, control method, and control program

Abstract

An electronic apparatus (100) comprises: an acquisition unit (131) for acquiring point cloud data (123) indicating a set of points that can identify road surface conditions; an identification unit (132) for identifying, in the detection range of the point cloud data (123), missing points in the point cloud data (123) which are below the height of the road surface; a supplementation unit (133) for supplementing the missing points with a pseudo point cloud at a position at a predetermined depth from the road surface; and a determination unit (135) for determining a depression in the road surface based on the point cloud data (123) that has been supplemented with the pseudo point cloud.

Description

[Technical Field]

[0001] The present application relates to an electronic device, a control method, and a control program. [Background technology]

[0002] There are known techniques for recognizing surrounding obstacles. Patent Document 1 discloses a technique for selecting points on the floor surface using plane parameters of the detected floor surface and recognizing obstacles based on these points. Patent Document 2 discloses a technique for calculating the degree of unevenness of the road surface based on input information from the road surface to the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-269937 [Patent Document 2] Japanese Patent Application Publication No. 2020-013537 Summary of the Invention [Problem to be solved by the invention]

[0004] It is desirable for autonomously moving vehicles to avoid not only surrounding objects but also small objects, steps, holes, etc. However, when using a camera mounted on a moving vehicle, it is difficult to determine recesses from images, and there is room for improvement in the accuracy of detecting recesses on a flat surface. [Means for solving the problem]

[0005] An electronic device according to one aspect includes an acquisition unit that acquires point cloud data corresponding to points on a road surface, a discrimination unit that discriminates missing points on the road surface that are below ground level and for which the point cloud data has not been acquired, a completion unit that complements a pseudo point cloud at a predetermined depth from the ground surface as a point cloud corresponding to the missing points, and a determination unit that determines depressions in the road surface based on the pseudo point cloud and the point cloud data.

[0006] An electronic device according to one aspect includes an acquisition unit that acquires point cloud data that indicates a set of points that can identify the condition of the road surface; a discrimination unit that discriminates missing portions of the point cloud data that are below the height of the road surface within the detection range of the point cloud data; a completion unit that complements the missing portions with a pseudo point cloud at a position at a predetermined depth from the road surface; and a determination unit that determines depressions in the road surface based on the point cloud data complemented with the pseudo point cloud.

[0007] A control method according to one aspect includes an acquisition process in which an electronic device acquires point cloud data corresponding to points on a road surface; a determination process in which an electronic device determines missing points on the road surface that are below ground level and for which point cloud data has not been acquired; a completion process in which an electronic device completes a pseudo point cloud at a predetermined depth from the ground surface as a point cloud corresponding to the missing points; and a determination process in which an electronic device determines depressions in the road surface based on the pseudo point cloud and the point cloud data.

[0008] A control program according to one aspect causes an electronic device to execute an acquisition process for acquiring point cloud data corresponding to points on a road surface; a determination process for determining missing points on the road surface that are below ground level and for which point cloud data has not been acquired; a completion process for completing a pseudo point cloud at a predetermined depth from the ground surface as a point cloud corresponding to the missing points; and a determination process for determining depressions in the road surface based on the pseudo point cloud and the point cloud data. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an overview of an electronic device according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of the electronic device according to the embodiment. [Figure 3] FIG. 3 is a flowchart illustrating an example of a processing procedure executed by the electronic device according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the detection data and the point cloud data. [Figure 5]FIG. 5 is a flowchart showing an example of the point cloud data generation process shown in FIG. [Figure 6] FIG. 6 is a flowchart showing an example of the cloth simulation shown in FIG. [Figure 7] FIG. 7 is a diagram for explaining the algorithm of the cloth simulation shown in FIG. [Figure 8] FIG. 8 is a diagram showing an example of a simulation result of the point cloud and the virtual cloth. [Figure 9] FIG. 9 is a diagram for explaining point cloud data used in the cross simulation shown in FIG. [Figure 10] FIG. 10 is a flowchart showing an example of the unevenness determination shown in FIG. [Figure 11] FIG. 11 is a diagram illustrating an example of the provided data of the electronic device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Several embodiments for implementing the electronic device, control method, control program, etc. according to the present application will be described in detail with reference to the drawings. Note that the present invention is not limited to the following description. Furthermore, the components in the following description include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. In the following description, similar components may be assigned the same reference numerals. Furthermore, duplicated descriptions may be omitted.

[0011] FIG. 1 is a diagram illustrating an overview of an electronic device according to an embodiment. The electronic device 100 shown in FIG. 1 is mounted on a moving body 1000 capable of at least one of autonomous driving and manual driving. The moving body 1000 includes, for example, a vehicle, a robot, a cart, a train, a flying object capable of taking off and landing, etc. The moving body 1000 is equipped with a detection unit 200 capable of detecting the surroundings in the moving direction. In FIG. 1, a direction Dx is the traveling direction of the moving body 1000. A direction Dy is the direction of gravity that intersects with the direction Dx and is the height direction.

[0012] The detection unit 200 includes, for example, a depth camera, a millimeter-wave radar, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), etc. The detection unit 200, for example, irradiates a pulsed laser beam and measures the reflected light of the laser beam to measure the distance, direction, etc. of an object. The detection unit 200 irradiates a road surface 5000 on which the mobile object 1000 is traveling with the laser beam, measures the reflected light reflected by the road surface 5000 and targets on the road surface 5000, etc., and supplies detection data indicating the measurement results of the detection range 310 to the electronic device 100, the mobile object 1000, etc. The detection range 310 includes a range in which detection of unevenness on the road surface 5000 is required. The detection range 310 includes a range in which detection of the path ahead of the mobile object 1000, the surrounding environment, etc. is required. The detection range 310 includes a range calculated or set based on road surface conditions such as the speed of the mobile object 1000 and the slipperiness of the road surface 5000. The climbable angle 320 includes an angle at which the moving body 1000 can climb a slope without scraping the bottom. If the moving body 1000 is a vehicle, the climbable angle 320 can be calculated from the height from the road surface 5000 to the bottom of the vehicle and the wheelbase width. The minimum altitude 330 includes an altitude that is reached when the climbable angle 320 continues throughout the entire detection range 310. The minimum altitude 330 can be calculated from the detection range 310 and the climbable angle 320. The moving body 1000 can recognize obstacles, targets, etc. based on the detection results of the detection unit 200.

[0013] The electronic device 100 is configured to be able to communicate with the mobile object 1000 wirelessly or via a wired connection, and obtains the measurement results of the detection unit 200 via the mobile object 1000. The electronic device 100 has a function of estimating the self-position of the mobile object 1000 and a surrounding map using the measurement results of the detection unit 200, for example, by a technique such as SLAM (Simultaneous Localization and Mapping). The surrounding map can be expressed as point cloud data, for example. The point cloud data has information on three-dimensional coordinates and colors. The electronic device 100 can estimate the likelihood of the estimated self-position based on the estimated surrounding map and the amount of movement of the mobile object 1000.

[0014] In the example shown in FIG. 1 , the electronic device 100 acquires measurement results obtained by the detection unit 200 measuring the area ahead in the traveling direction M of the mobile object 1000. In this case, a slope 5100 exists on a road surface 5000 in the traveling direction M of the mobile object 1000. The electronic device 100 may determine that the slope 5100 is a "drivable road surface" because it is a slope at an angle that allows it to be climbed. The electronic device 100 may detect any depressions on the road surface 5000 or the slope 5100 that are large enough for a tire to get stuck in. If the depression is deep, the electronic device 100 may be unable to measure the reflected light, resulting in a defect, and may determine that the depression is highly likely to be a dangerous location for the mobile object 1000.

[0015] The road surface 5000 includes objects such as the surface of a road or a floor, over which the mobile object 1000 moves. The electronic device 100 acquires point cloud data capable of identifying the condition of the road surface 5000 from the measurement results obtained by the detection unit 200. If the slope 5100 of the road surface 5000 is deep, the detection unit 200 may not be able to measure reflected light. In this case, the point cloud data acquired by the electronic device 100 will be data lacking point clouds in deep parts of the slope 5100. In other words, deep parts of the slope 5100 are likely to be holes or the like in the road surface 5000, and are therefore likely to be dangerous areas on the road surface 5000 for the mobile object 1000 to move in. For this reason, the electronic device 100 according to this embodiment provides a technology for improving the accuracy of determining the slope 5100 around the mobile object 1000.

[0016] [Example of electronic device configuration] Fig. 2 is a diagram showing an example of the configuration of the electronic device 100 according to the embodiment. As shown in Fig. 2, the electronic device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The control unit 130 is electrically connected to the communication unit 110, the storage unit 120, etc. The electronic device 100 may also include other configurations.

[0017] The communication unit 110 can communicate with, for example, the mobile object 1000, other communication devices, etc. The communication unit 110 can support various communication standards. The communication unit 110 can send and receive various types of data via, for example, a wired or wireless network. The communication unit 110 can provide the received data to the control unit 130. The communication unit 110 can send data to a destination instructed by the control unit 130.

[0018] The storage unit 120 can store programs and data. The storage unit 120 is also used as a working area for temporarily storing processing results of the control unit 130. The storage unit 120 may include any non-transitory storage medium, such as a semiconductor storage medium or a magnetic storage medium. The storage unit 120 may include multiple types of storage media. The storage unit 120 may include a combination of a portable storage medium, such as a memory card, an optical disk, or a magneto-optical disk, and a storage medium reader. The storage unit 120 may include a storage device used as a temporary storage area, such as a RAM (Random Access Memory).

[0019] The storage unit 120 can store various data such as a program 121, a cloth simulation 122, point cloud data 123, posture data 124, and provided data 125. The program 121 causes the control unit 130 to execute a function of determining depressions present on a road surface 5000, a slope 5100, or the like. The cloth simulation 122 is a program capable of executing a cloth simulation method that estimates the shape of the point cloud data 123 from the shape of a cloth that has a predetermined tension and is placed over the point cloud data 123 under a predetermined gravity, and outputs the estimation result. The cloth simulation 122 may be included in the program 121 or may be stored in an external storage device or the like. The point cloud data 123 is data that indicates the shape of the road surface 5000, etc. detected by the detection unit 200, as a point cloud. The posture data 124 is data that can identify the posture of the detection unit 200 or the moving object 1000. The posture data 124 is data acquired at a predetermined timing, for example, at the time of setup or the start of processing. The provided data 125 includes the determination result made by the electronic device 100, and data that can identify depressions present on the road surface 5000, the slope 5100, and the like.

[0020] The control unit 130 is an arithmetic processing device. Examples of the arithmetic processing device include, but are not limited to, a central processing unit (CPU), a system-on-a-chip (SoC), a micro control unit (MCU), a field-programmable gate array (FPGA), and a coprocessor. The control unit 130 can comprehensively control the operation of the electronic device 100 to realize various functions.

[0021] Specifically, the control unit 130 can execute instructions included in the program 121 stored in the storage unit 120 while referring to the data stored in the storage unit 120 as needed. The control unit 130 then controls the functional units in accordance with the data and instructions, thereby realizing various functions. The functional units include, but are not limited to, a communication unit 230, for example.

[0022] The control unit 130 has functional units such as an acquisition unit 131, a discrimination unit 132, a complement unit 133, a cross simulation unit 134, a determination unit 135, and a provision unit 136. The control unit 130 executes the program 121 to realize the functions of the acquisition unit 131, the discrimination unit 132, the complement unit 133, the cross simulation unit 134, the determination unit 135, and the provision unit 136. The program 121 is a program for causing the control unit 130 of the electronic device 100 to function as the acquisition unit 131, the discrimination unit 132, the complement unit 133, the cross simulation unit 134, the determination unit 135, and the provision unit 136.

[0023] The acquisition unit 131 acquires point cloud data 123 that indicates a collection of points that can identify the state of the road surface 5000. The acquisition unit 131 acquires the point cloud data 123 generated based on a depth image of the detection data from the detection unit 200, posture data of the moving object 1000, etc. The acquisition unit 131 may acquire point cloud data 123 that is generated outside the electronic device 100. The acquisition unit 131 stores the acquired point cloud data 123 in the storage unit 120.

[0024] The discrimination unit 132 discriminates missing portions of the point cloud data 123 that are at or below the height of the road surface 5000 within the detection range 310 of the point cloud data 123. The missing portions of the point cloud data 123 are recessed portions of the road surface 5000 or the slope 5100. The discrimination unit 132 discriminates whether or not a point cloud is missing in the point cloud data 123. The discrimination unit 132 associates missing data that can identify missing portions in the point cloud data 123 with the point cloud data 123.

[0025] The complementing unit 133 complements missing portions of the point cloud data 123 with pseudo point clouds at positions at a predetermined depth from the road surface 5000. The complementing unit 133 calculates the minimum altitude 330 calculated from the detection range 310 of the point cloud data 123 and the climbable angle 320 of the mobile object 1000. The complementing unit 133 complements missing portions with pseudo point clouds at heights lower than the calculated minimum altitude 330. The complementing unit 133 complements missing portions with pseudo point clouds at a depth that can be sufficiently regarded as depressions in the road surface 5000 or the slope 5100.

[0026] The cloth simulation unit 134 outputs the shape of the point cloud data 123 from the cloth shape when a virtual cloth with a predetermined tension is draped over the point cloud data 123 under a predetermined gravity. By executing the cross simulation 122, the cloth simulation unit 134 connects, among the multiple points included in the point cloud indicated by the point cloud data 123, a first point that is the highest in the height direction and a second point that is around the first point and is lower in the height direction (direction Dy shown in FIG. 1 ), so as to form a bendable cloth shape, thereby outputting the shape of the point cloud data 123. By executing the cross simulation 122, the cloth simulation unit 134 may also output the shape of the point cloud data 123 by connecting, among the multiple points included in the point cloud indicated by the point cloud data 123, a certain point and its surrounding points so as to form a bendable cloth shape.

[0027] The determination unit 135 determines depressions on the road surface 5000 and the slope 5100 based on the point cloud data 123 that has been supplemented with the pseudo point cloud. The determination unit 135 uses the shape of the point cloud data 123 output by the cross simulation unit 134 and the supplemented pseudo point cloud to determine depressions on the road surface 5000 and the slope 5100. The determination unit 135 associates the determination result of the slope 5100 and determination data that indicates the area of ​​the slope 5100 in the point cloud data 123 with the point cloud data 123 and stores them in the storage unit 120.

[0028] The providing unit 136 provides provided data 125 capable of identifying depressions on the road surface 5000 or the slope 5100 determined by the determining unit 135. The providing unit 136 can provide the provided data 125 to the moving object 1000 via the communication unit 110. This enables the moving object 1000 to change its route so as to avoid the depressions on the road surface 5000 or the slope 5100 indicated by the provided data 125. The providing unit 136 may provide the provided data 125 to a display device, a smartphone, or the like.

[0029] An example of the functional configuration of the electronic device 100 according to this embodiment has been described above. Note that the configuration described above using Fig. 2 is merely an example, and the functional configuration of the electronic device 100 according to this embodiment is not limited to this example. The functional configuration of the electronic device 100 according to this embodiment can be flexibly modified according to specifications and operations.

[0030] [Example of disposal procedure for electronic devices] FIG. 3 is a flowchart showing an example of a processing procedure executed by the electronic device 100 according to the embodiment. FIG. 4 is a diagram showing an example of detection data and point cloud data. FIG. 5 is a flowchart showing an example of a process for generating the point cloud data 123 shown in FIG. 3. FIG. 6 is a flowchart showing an example of the cloth simulation shown in FIG. 3. FIG. 7 is a diagram for explaining an algorithm of the cross simulation shown in FIG. 6. FIG. 8 is a diagram showing an example of a simulation result of the point cloud 123P and the virtual cloth 40. FIG. 9 is a diagram for explaining point cloud data used in the cross simulation shown in FIG. 6. FIG. 10 is a flowchart showing an example of the unevenness determination shown in FIG. 3. The processing procedures shown in FIGS. 3, 5, 6, and 10 are implemented by the control unit 130 of the electronic device 100 executing the program 121.

[0031] As shown in FIG. 3, the control unit 130 of the electronic device 100 acquires detection data 210 from the detection unit 200 (step S100). For example, the control unit 130 acquires the detection data 210 from the detection unit 200 of the moving object 1000 via the communication unit 110. In this embodiment, the control unit 130 acquires the detection data 210 having a depth image capturing an image of the area ahead of the moving object 1000, as shown in FIG. 4. In the example shown in FIG. 4, the detection data 210 has a depth image capturing an incline 5100 on a road surface 5000 near the moving object 1000 and two convex portions 5200 on the road surface 5000 ahead of the incline 5100. Returning to FIG. 3, the control unit 130 stores the acquired detection data 210 in the storage unit 120. If the control unit 130 is unable to acquire the detection data 210, the control unit 130 stores in the storage unit 120 a message indicating that the detection data 210 could not be acquired. When the process of step S100 ends, control unit 130 advances the process to step S200.

[0032] The control unit 130 determines whether the acquisition of the detection data 210 was successful (step S200). For example, if the storage unit 120 stores the latest detection data 210, the control unit 130 determines that the acquisition of the detection data 210 was successful. If the control unit 130 determines that the acquisition of the detection data 210 was not successful (No in step S200), the control unit 130 ends the processing procedure shown in Fig. 3. Note that the processing procedure shown in Fig. 3 may be a processing procedure in which, if the acquisition of the detection data 210 was not successful, the processing to acquire the detection data 210 again in step S100 is executed.

[0033] Furthermore, if the control unit 130 determines that the acquisition of the detection data 210 has been successful (Yes in step S200), the control unit 130 proceeds to step S300. The control unit 130 executes a process for generating the point cloud data 123 (step S300). For example, the control unit 130 converts the depth image indicated by the acquired detection data 210 into the point cloud data 123 by performing a known coordinate transformation on the depth image.

[0034] For example, as shown in FIG. 5, the control unit 130 generates a point cloud in the camera coordinate system from pixel values ​​and pixel coordinates (x, y) of the depth image (step S301). Then, the control unit 130 calculates a rotation matrix of the point cloud data 123 from the orientation data 124 stored in the storage unit 120 and converts the point cloud data 123 to a world coordinate system (step S302). The world coordinate system is one of the coordinate systems used in fields such as three-dimensional computer graphics. To represent the position and movement of an object in three-dimensional space, the entire space is defined as a coordinate system using X, Y, and Z axes. The control unit 130 then removes point clouds outside the range and noise from the point cloud data 123 (step S303). For example, the control unit 130 reduces the number of point clouds to be processed by removing point clouds outside the detection range 310 and noise. In the example shown in FIG. 4, the point cloud data 123 is a perspective view of a road surface 5000 having a slope 5100 and two convex portions 5200. In this case, the point cloud data 123 is missing the point cloud of the slope 5100. Returning to Fig. 5, the control unit 130 stores the generated point cloud data 123 in the storage unit 120 (step S304). When the processing procedure shown in Fig. 5 ends, the control unit 130 returns to step S300 shown in Fig. 3 and proceeds to step S400.

[0035] The control unit 130 executes a cross simulation (step S400). For example, the control unit 130 executes the cross simulation 122 in the storage unit 120 to perform a cross simulation on the point cloud data 123.

[0036] For example, as shown in Fig. 6, the control unit 130 generates a virtual cloth and associates the point cloud with the cloth grid points (step S401). The virtual cloth is a virtual cloth of a size that can cover the point cloud of the point cloud data 123, and has a plurality of grid points arranged in a matrix. After generating the virtual cloth, the control unit 130 sets the surface hardness of the point cloud corresponding to each cloth grid point of the virtual cloth.

[0037] The control unit 130 determines a missing portion 123D of the point cloud data 123 (step S402). For example, when the point cloud of the point cloud data 123 is covered with a virtual cloth, the control unit 130 determines a portion where the cloth grid points of the virtual cloth are not linked to the point cloud as the missing portion 123D.

[0038] For example, as shown in scene C0 in Fig. 7, the point cloud data 123 has a plurality of point clouds 123P. For simplicity of explanation, Fig. 7 shows a cross section of a portion of the plurality of point clouds 123P of the point cloud data 123. In this case, the control unit 130 places the virtual cloth 40 so as to cover the plurality of point clouds 123P, and associates the cloth grid points 400 with the point clouds 123P of the point cloud data 123. For example, when the virtual cloth 40 is brought into contact with the point cloud data 123, the control unit 130 determines that a portion of the cloth grid point 400a that is not in contact with the point cloud 123P is a missing portion 123D of the point cloud data 123.

[0039] Returning to FIG. 6, the control unit 130 associates the determination result with the point cloud data 123 and stores it in the storage unit 120, and then proceeds to step S403. The control unit 130 executes a process of interpolating the point cloud data 123 (step S403). For example, the control unit 130 executes an interpolation process to interpolate the pseudo point cloud 123S into the missing portion 123D of the point cloud data 123. For example, when the missing portion 123D of the point cloud data 123 has been determined, the control unit 130 determines the height of the point cloud to be interpolated based on the detection range 310, the climbable angle 320, and the minimum altitude 330 shown in FIG. 1. The height of the point cloud to be interpolated is determined based on detection requirements, vehicle requirements, etc., so that it is lower than the minimum altitude 330.

[0040] For example, as shown in scene C0 in Fig. 7, when the point cloud data 123 has a missing portion 123D, the control unit 130 determines the height of the point cloud to be interpolated into the missing portion 123D, and interpolates the point cloud data 123 with a pseudo point cloud 123S at that height. As a result, as shown in scene C1 in Fig. 7, the control unit 130 interpolates the point cloud data 123 with a pseudo point cloud 123S of a depth that allows the missing portion 123D to be sufficiently determined as a slope 5100.

[0041] 6, the control unit 130 advances the process to step S404 after complementing the missing portion 123D of the point cloud data 123 with the pseudo point cloud 123S. Note that, if it is determined in the above-described step S402 that the missing portion 123D does not exist in the point cloud data 123, the control unit 130 skips the process of step S403 and advances the process to step S404.

[0042] The control unit 130 reflects the external force (gravity) on the virtual cloth 40 (step S404). For example, as shown in scene C2 of FIG. 7, the control unit 130 moves the cloth grid points 400 of the virtual cloth 40 from the starting point 410 in the direction M1 (gravity direction) below the point cloud 123P. Then, as shown in scene C3 of FIG. 7, when the cloth grid points 400 of the virtual cloth 40 become lower than the point cloud 123P, the control unit 130 moves the cloth grid points 400A of the virtual cloth 40 that are associated with the point cloud 123P that indicates the road surface 5000 from the starting point 410 in the direction M2 to above the point cloud 123P. In this case, the control unit 130 does not move the cloth grid points 400 of the virtual cloth 40 that are not associated with the point cloud 123P that indicates the road surface 5000. Returning to FIG. 6, when the process of step S404 ends, control unit 130 advances the process to step S405.

[0043] The control unit 130 reflects the internal force (tension) on the virtual cloth 40 (step S405). For example, the control unit 130 reflects the pulling force between the multiple cloth grid points 400 of the virtual cloth 40. The tension is fixed at a value that allows the slope 5100, which is the object to be detected, to be detected. For example, as shown in scene C4 of FIG. 7, the control unit 130 moves the cloth grid points 400 of the virtual cloth 40 that are not linked to the point cloud 123P from the start point 410 toward the direction M2 according to the tension. As a result, the control unit 130 obtains the cloth grid points 400B of the virtual cloth 40 from one side (the sky side) that are not attached to the point cloud 123P, i.e., the cloth grid points 400B that are not attached to the road surface 5000 indicated by the point cloud 123P. The control unit 130 distinguishes the multiple cloth grid points 400 into cloth grid points 400A that are attached to the road surface 5000 and cloth grid points 400B that are not attached to the road surface 5000. Returning to FIG. 6, when the processing of step S405 ends, the control unit 130 advances the processing to step S406.

[0044] The control unit 130 determines whether the predetermined number of times has been reached (step S406). For example, in the case of a steep slope or a wall, the virtual cloth 40 may not fall all the way down to the road surface 5000 during cloth simulation. For this reason, the control unit 130 sets a predetermined number of times for executing cloth simulation from two directions, and determines that the predetermined number of times has been reached when the number of executions matches the predetermined number of times. If the control unit 130 determines that the predetermined number of times has not been reached (No in step S406), the control unit 130 returns the process to step S404, which has already been described, and continues the process. If the control unit 130 determines that the predetermined number of times has been reached (Yes in step S406), the control unit 130 proceeds to step S407.

[0045] The control unit 130 pastes the virtual cloth 40, of the virtual cloth 40 that is not pasted on the road surface 5000, at a location that has no elevation difference from the location already pasted (step S407). For example, assume that the control unit 130 has obtained a simulation result of the point cloud 123P and the virtual cloth 40, as shown in FIG. 8. In this case, the control unit 130 determines that the cloth grid point 400B-2 adjacent to the already fixed cloth grid point 400A-1 may be fixed to the actual measurement value because the point clouds 123P-1 and 123P-2 indicating the heights of the road surfaces being compared are almost the same height. The control unit 130 determines that the cloth grid point 400B-4 adjacent to the already fixed cloth grid point 400A-3 should not be fixed to the actual measurement value because there is a difference between the point clouds 123P-3 and 123P-4 indicating the heights of the road surfaces being compared. Returning to FIG. 6, once the control unit 130 stores the processing results for the underground side of the road surface 500 in the storage unit 120, the control unit 130 advances the processing to step S408.

[0046] The control unit 130 determines whether the specified number of times has been reached (step S408). If the control unit 130 determines that the specified number of times has not been reached (No in step S408), the process returns to step S407, which has already been described, and the process continues. If the control unit 130 determines that the specified number of times has been reached (Yes in step S408), the process proceeds to step S409.

[0047] The control unit 130 flips the point cloud data 123 upside down (step S409). For example, the control unit 130 flips the sky side and the underground side of the point cloud 123P (road surface 5000) indicated by the point cloud data 123. When the process of step S409 ends, the control unit 130 advances the process to step S410.

[0048] The control unit 130 reflects the external force (gravity) on the virtual cloth 40 (step S410). For example, similar to step S404 already described, the control unit 130 performs a process of overlaying the cloth grid points 400 of the virtual cloth 40 onto the point cloud data 123 obtained by inverting the point cloud 123P and the pseudo point cloud 123S, as shown in Fig. 9. Returning to Fig. 6, when the process of step S410 ends, the control unit 130 advances the process to step S411.

[0049] The control unit 130 reflects the internal force (tension) in the virtual cloth 40 (step S411). For example, the control unit 130 reflects the pulling force between the multiple cloth grid points 400 of the virtual cloth 40, as in step S405 already described. As a result, the control unit 130 obtains a result in which, among the cloth grid points 400 of the virtual cloth 40 from the other side (underground side), the points that are not attached to the point cloud 123P, i.e., the points that are not attached to the road surface 5000 indicated by the point cloud 123P. When the process of step S411 ends, the control unit 130 proceeds to step S412.

[0050] The control unit 130 determines whether the predetermined number of times has been reached (step S412). For example, similar to the already-described step S408, if the control unit 130 determines that the predetermined number of times has not been reached (No in step S412), the control unit 130 returns the process to the already-described step S410 and continues the process. On the other hand, if the control unit 130 determines that the predetermined number of times has been reached (Yes in step S412), the control unit 130 proceeds to step S413.

[0051] The control unit 130 pastes the virtual cloth 40 that is not attached to the road surface 5000 at a location that has no difference in elevation with the already attached location (step S413). The control unit 130 makes the same determination as in step S407 already described, and stores the processing result for the sky side of the road surface 5000 in the storage unit 120, and then proceeds to step S414.

[0052] The control unit 130 determines whether the predetermined number of times has been reached (step S414). If the control unit 130 determines that the predetermined number of times has not been reached (No in step S414), the process returns to step S413, which has already been described, and continues the process. If the control unit 130 determines that the predetermined number of times has been reached (Yes in step S414), the control unit 130 ends the process procedure shown in Fig. 6, returns to step S400 shown in Fig. 3, and proceeds to step S500.

[0053] The control unit 130 determines whether the point cloud 123P in the point cloud data 123 is uneven (step S500). For example, the control unit 130 determines whether the point cloud 123P is uneven by using a determination condition. The determination condition includes a condition that determines the point cloud 123P as a road surface 5000 if the difference in elevation between the virtual cloth 40 on the sky side and the point cloud data 123 is equal to or less than a specified value, and determines the point cloud 123P as a depression if the difference in elevation between the virtual cloth 40 on the sky side and the point cloud data 123 is not equal to or less than the specified value. The determination condition also includes a condition that determines the point cloud 123P as a road surface 5000 if the difference in elevation between the virtual cloth 40 on the underground side and the point cloud data 123 is equal to or less than a specified value, and determines the point cloud 123P as a convexity if the difference in elevation between the virtual cloth 40 on the underground side and the point cloud data 123 is not equal to or less than a specified value. Then, the control unit 130 clusters the point cloud 123P separated from the road surface 5000, and then performs the following process to determine whether each cluster is uneven, uneven, or convex. A cluster is, for example, a collection of points in the point cloud 123P that are not determined to be part of the road surface 5000, such as depressions and projections.

[0054] In the case of a single piece of cloth, the control unit 130 may separate the concave portions between the virtual cloth 40 on the sky side and the point cloud 123P, and the convex portions between the virtual cloth 40 on the underground side and the point cloud 123P. An example of the process of determining the concave and convex portions is shown below.

[0055] As shown in FIG. 10, the control unit 130 acquires the simulation results for the underground side (step S501). The control unit 130 executes a convex extraction loop process (step S502). The termination condition for the convex extraction loop process includes, for example, execution for all point clouds. After executing the convex extraction loop process, the control unit 130 determines whether the elevation difference between the virtual cloth 40 and the ground surface is equal to or greater than a specified value (step S503). For example, the control unit 130 calculates the elevation difference between the cloth grid points 400 of the virtual cloth 40 and the ground surface indicated by the point cloud 123P, and compares the elevation difference with the specified value. If the control unit 130 determines that the elevation difference is equal to or greater than the specified value (Yes in step S503), the control unit 130 proceeds to step S504.

[0056] The control unit 130 determines that the point group 123P is convex (step S504). For example, the control unit 130 determines that the extraction target point group 123P is convex, and associates data that can identify convexities with the point group 123P. When the process of step S504 ends and the termination condition of the extraction loop process of step S502 is not satisfied, the control unit 130 continues the convex extraction loop process for the next point group 123P. When the process of step S504 ends and the termination condition of the convex extraction loop process of step S502 is satisfied, the control unit 130 advances the process to step S505, which will be described later.

[0057] Furthermore, if control unit 130 determines in step S503 that the elevation difference is not equal to or greater than a specified value (No in step S503), and if the termination condition for the extraction loop processing in step S502 is not satisfied, control unit 130 continues the convex extraction loop processing for the next point cloud 123P. Furthermore, if control unit 130 determines that the elevation difference is not equal to or greater than a specified value (No in step S503), and if the termination condition for the convex extraction loop processing in step S502 is satisfied, control unit 130 proceeds to the processing in step S505.

[0058] The control unit 130 clusters the point group determined to be convex (step S505). For example, the control unit 130 clusters the convex point group 123P based on the determination result of the extraction loop processing in step S502. When the processing of step S505 ends, the control unit 130 proceeds to the processing of step S506.

[0059] The control unit 130 acquires the simulation results for the sky side (step S506). The control unit 130 executes a concave extraction loop process (step S507). The termination condition for the concave extraction loop process includes, for example, execution for all point clouds. After executing the concave extraction loop process, the control unit 130 determines whether the elevation difference between the virtual cloth 40 and the ground surface is equal to or greater than a specified value (step S508). For example, the control unit 130 calculates the elevation difference between the cloth grid points 400 of the virtual cloth 40 and the ground surface indicated by the point cloud 123P, and compares the elevation difference with a specified value. If the control unit 130 determines that the elevation difference is equal to or greater than the specified value (Yes in step S508), the control unit 130 proceeds to step S509.

[0060] Control unit 130 determines that point group 123P is a concave (step S509). For example, control unit 130 determines that point group 123P to be extracted is a concave, and associates data that can identify a concave with point group 123P. When the process of step S509 ends and the termination condition of the concave extraction loop process of step S507 is not satisfied, control unit 130 continues the concave extraction loop process for the next point group 123P. When the process of step S509 ends and the termination condition of the extraction loop process of step S502 is satisfied, control unit 130 proceeds to the process of step S510, which will be described later.

[0061] Furthermore, if control unit 130 determines that the elevation difference is not equal to or greater than the specified value (No in step S508), and if the termination condition for the extraction loop processing in step S507 is not satisfied, it executes the concave extraction loop processing for the next point cloud 123P. Furthermore, if control unit 130 determines that the elevation difference is not equal to or greater than the specified value (No in step S508), and if the termination condition for the extraction loop processing in step S507 is satisfied, it proceeds to step S510.

[0062] The control unit 130 clusters the point group determined to be concave (step S510). For example, the control unit 130 clusters the concave point group 123P based on the determination result of the extraction loop processing in step S507. When the processing of step S510 ends, the control unit 130 advances the processing to step S511.

[0063] The control unit 130 determines that the point cloud 123P that has not been determined to be a convex or concave portion is the road surface 5000 (step S511). For example, the control unit 130 determines that the point cloud 123P that has not been determined to be a convex or concave portion is the road surface 5000 based on the point cloud data 123 and the processing results of steps S502 and S507, and associates the determination result with the point cloud data 123. When the processing of step S511 ends, the control unit 130 ends the processing procedure shown in Fig. 10, returns to step S500 shown in Fig. 3, and proceeds to step S600.

[0064] The control unit 130 provides the provided data 125 indicating the determination result (step S600). For example, the control unit 130 generates the provided data 125 capable of identifying the slope 5100 in the point cloud data 123, and executes a process of providing the provided data 125. The providing process includes, for example, a process of providing (transmitting) the provided data 125 to the mobile object 1000 via the communication unit 110, a process of providing (transmitting) the provided data 125 to another electronic device via the communication unit 110, a process of displaying the provided data on a display device via the communication unit 110, etc.

[0065] FIG. 11 is a diagram illustrating an example of the provided data 125 of the electronic device 100 according to the embodiment. As illustrated in FIG. 11, the control unit 130 generates the provided data 125 indicating the results of determining unevenness for the point cloud data 123 illustrated in FIG. 4. The provided data 125 is data that enables identification of the determination results between the road surface 5000, the recessed portion 5100a, and the convex portion 5200. By providing the provided data 125 that enables identification of the recessed portion 5100a and the convex portion 5200 on the road surface 5000, the control unit 130 can assist in improving the accuracy of detecting the surrounding recessed portion 5100a. This allows the electronic device 1000 to accurately identify the recessed portion 5100a on the road surface 5000 and move while avoiding the recessed portion 5100a, thereby assisting in improving safety. Furthermore, the electronic device 100 can accurately detect the recessed portion 5100a even on the road surface 5000 on which the mobile object 1000 is not traveling, thereby contributing to improved safety.

[0066] Returning to FIG. 3, when the processing of step S600 ends, control unit 130 determines whether to end (step S700). For example, control unit 130 determines to end if an end instruction has been received, it is time to end, or the like. If control unit 130 determines not to end (No in step S700), it returns the processing to step S100 already described and continues the processing. On the other hand, if control unit 130 determines to end (Yes in step S700), it ends the processing procedure shown in FIG. 3.

[0067] As described above, the electronic device 100 can use the point cloud data 123 representing the road surface 5000 to determine the missing portion 123D of the point cloud data 123 that is below the height of the road surface, and can complement the missing portion 123D with the pseudo point cloud 123S at a position at a predetermined depth from the road surface 5000. The electronic device 100 can determine the slope 5100 of the road surface based on the point cloud data 123 complemented with the pseudo point cloud 123S. As a result, even if the slope 5100 of the road surface 5000 cannot be converted into a point cloud, the electronic device 100 can perform a cross simulation by complementing the pseudo point cloud 123S. As a result, the electronic device 100 can improve the accuracy of detecting the depression 5100a on the road surface 5000, the slope 5100, etc., even when the point cloud data 123 is missing the point cloud 123P of the deep slope 5100, etc.

[0068] The electronic device 100 outputs the shape of the virtual cloth 40 when the virtual cloth 40 with a predetermined tension is placed over the point cloud data 123 under a predetermined gravity, and can determine the depressions 5100a on the road surface 5000, the slope 5100, etc. by using the pseudo point cloud 123S interpolated with the shape of the virtual cloth 40. This allows the electronic device 100 to improve the accuracy of detecting the depressions 5100a on the road surface 5000, the slope 5100, etc. even when the actually measured point cloud 123P is missing and cannot be linked to the cloth grid points 400 of the virtual cloth 40.

[0069] When the cloth simulation unit 134 brings the cloth grid points 400 of the virtual cloth 40 closer to the point cloud 123P indicated by the point cloud data 123 under a predetermined gravity, the electronic device 100 determines whether to fix the cloth grid points 400 to the point cloud 123P so that a curved cloth shape is formed between a first point that is the highest in the height direction among the points included in the point cloud 123P and a second point that is around the first point and lower in the height direction than the first point, thereby being able to output the shape of the virtual cloth 40. As a result, even if the slope 5100 is a steep slope or a wall, the virtual cloth 40 falls all the way down to the point cloud 123P that indicates the road surface 5000, and the electronic device 100 can accurately determine the shape of the slope 5100.

[0070] The electronic device 100 acquires point cloud data 123 based on a depth image obtained by the detection unit 200 of the moving body 1000 measuring the road surface 5000, and sets the height of the pseudo point cloud 123S based on the detection range 310 of the point cloud data 123 (depth image), the climbable angle 320 of the moving body 1000, and the minimum altitude 330 calculated from the detection range 310 and the climbable angle 320. This allows the electronic device 100 to supplement the point cloud data 123 with the pseudo point cloud 123S that is appropriate for the attitude of the moving body 1000, the detection unit 200, etc., and therefore allows the slope 5100 to be determined more accurately.

[0071] In the present embodiment, the electronic device 100 covers the point cloud 123P with the virtual cloth 40 from the sky side and the underground side, but the present invention is not limited to this. For example, the electronic device 100 may use two virtual cloths 40 to cover the point cloud 123P from the sky side and the underground side.

[0072] In the present embodiment, the electronic device 100 is described as being mounted on the mobile object 1000, but the present invention is not limited to this. For example, the electronic device 100 may be implemented as a server, a roadside device, or the like, outside the mobile object 1000.

[0073] The electronic device of the present disclosure may include an acquisition unit that acquires point cloud data corresponding to points on a road surface, a discrimination unit that discriminates missing points that are points below the ground level of the road surface and for which no point cloud data has been acquired, a completion unit that complements pseudo point clouds at positions at a predetermined depth from the ground surface as point clouds corresponding to the missing points, and a determination unit that determines depressions in the road surface based on the pseudo point clouds and the point cloud data. In Figure 1, a horizontal line representing the road surface 5000 may be the ground surface of the road surface. The ground surface may be a straight line extending horizontally to the ground surface on which the mobile object is currently placed.

[0074] Characteristic embodiments have been described to fully and clearly disclose the technology claimed in the appended claims. However, the appended claims should not be limited to the above-described embodiments, but should be construed to embody all modifications and alternative configurations that may be conceived by those skilled in the art within the scope of the basic concepts set forth herein. Those skilled in the art can make various modifications and alterations to the content of the present disclosure based on the present disclosure. Therefore, these modifications and alterations are within the scope of the present disclosure. For example, in each embodiment, each functional unit, each means, each step, etc. may be added to other embodiments without logical contradiction, or may be replaced with each functional unit, each means, each step, etc. of other embodiments. Furthermore, in each embodiment, multiple functional units, each means, each step, etc. may be combined or divided into one. Furthermore, each embodiment of the present disclosure described above is not limited to faithful implementation of each described embodiment, and each feature may be combined or partially omitted as appropriate. The method of the present disclosure may be implemented by a device equipped with a CPU and memory, in which the CPU executes a program stored in the memory. [Explanation of symbols]

[0075] 40 Virtual Cloth 100 Electronic equipment 110 Communications Department 120 Storage section 121 Programs 122 Cloth Simulation 123 point cloud data 123D Missing part 123P point cloud 123S pseudo point cloud 124 posture data 125 Provided Data 130 Control Unit 131 Acquisition Department 132 Discrimination part 133 Complementary Section 134 Cross Simulation Section 135 Judgment section 136 Provision Department 200 Detector 310 Detection Range 320 Climbing angle 330 minimum altitude 400 cloth grid points 1000 Mobile 5000 road surface 5100 Slope 5100a Recess 5200 convex part

Claims

1. an acquisition unit that acquires point cloud data corresponding to points on a road surface; a discrimination unit that discriminates defective portions on the road surface that are located below the ground level of the road surface and for which the point cloud data has not been acquired; a completion unit that completes a pseudo point cloud at a position at a predetermined depth from the ground surface as a point cloud corresponding to the missing portion; a determination unit that determines depressions in the road surface based on the pseudo point cloud and the point cloud data; An electronic device comprising:

2. an acquisition unit that acquires point cloud data that indicates a set of points that can identify the condition of a road surface; a discrimination unit that discriminates missing portions of the point cloud data that are below the height of the road surface within a detection range of the point cloud data; a complementing unit that complements the missing portion with a pseudo point cloud at a position at a predetermined depth from the road surface; a determination unit that determines depressions in the road surface based on the point cloud data obtained by complementing the pseudo point cloud; An electronic device comprising:

3. a cloth simulation unit that outputs a shape of a virtual cloth having a predetermined tension and a predetermined gravity when the virtual cloth is placed over the point cloud data, the determination unit determines depressions in the road surface by using the shape of the virtual cloth output by the cloth simulation unit and the pseudo point cloud complemented therewith.

3. The electronic device according to claim 1 or 2.

4. the cloth simulation unit outputs the shape of the virtual cloth by determining whether or not to fix the cloth grid points to the point cloud so that, when the cloth grid points of the virtual cloth are brought close to the point cloud indicated by the point cloud data under a predetermined gravity, a curved cloth shape is formed between a first point that is the highest in a height direction among the points included in the point cloud and a second point that is around the first point and is lower in the height direction than the first point. The electronic device according to claim 3 .

5. the acquisition unit acquires the point cloud data based on a depth image obtained by a detection unit mounted on a moving body measuring the road surface; the height of the pseudo point cloud is set based on a detection range of the point cloud data, a climbable angle of the moving object, and a minimum altitude calculated from the detection range and the climbable angle; 5. The electronic device according to claim 4.

6. a providing unit that provides provided data that can identify the depressions in the road surface determined by the determining unit, The electronic device according to claim 5 .

7. the providing unit provides the provided data to the mobile object.

7. The electronic device according to claim 6.

8. the acquisition unit acquires attitude data of the moving body; the complementing unit complements the missing portion with a pseudo point cloud at a position at a predetermined depth from the road surface based on the posture data.

8. The electronic device according to claim 7.

9. Electronic devices, an acquisition step of acquiring point cloud data corresponding to points on a road surface; a discrimination step of discriminating defective portions on the road surface that are located below the ground level of the road surface and for which the point cloud data has not been acquired; a complementing step of complementing a pseudo point cloud at a position at a predetermined depth from the ground surface as a point cloud corresponding to the missing portion; a determination step of determining depressions in the road surface based on the pseudo point cloud and the point cloud data; A control method comprising:

10. For electronic devices, an acquisition step of acquiring point cloud data corresponding to points on a road surface; a discrimination step of discriminating defective portions on the road surface that are located below the ground level of the road surface and for which the point cloud data has not been acquired; a complementing step of complementing a pseudo point cloud at a position at a predetermined depth from the ground surface as a point cloud corresponding to the missing portion; a determination step of determining depressions in the road surface based on the pseudo point cloud and the point cloud data; A control program that executes the above.

Images