Information processing device, information processing system, information processing method, and program

Abstract

This information processing device comprises a first acquisition unit, a second acquisition unit, an imparting unit, and an output unit. The first acquisition unit generates, on the basis of radar information, a two-dimensional point cloud obtained by converting a three-dimensional point cloud. The second acquisition unit acquires, on the basis of a captured image captured using light, a two-dimensional captured image represented using a coordinate system shared with the two-dimensional point cloud, and label information imparted to an object included in the two-dimensional captured image. The imparting unit imparts the label information to the two-dimensional point cloud on the basis of the positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image. The output unit outputs output information in which label information is imparted to a point cloud based on the radar information.

Description

Information processing device, information processing system, information processing method, and recording medium

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and a recording medium.

[0002] For example, Patent Document 1 discloses a point cloud information processing device for improving robustness in aligning multiple pieces of point cloud information. According to Patent Document 1, the point cloud information processing device includes an image analysis unit, a point cloud label assignment unit, and a point cloud integration unit.

[0003] The image analysis unit analyzes image information captured from different viewpoints, recognizes different regions in each image, labels each region, and generates labeled image information. The point cloud labeling unit generates labeled point cloud information by assigning a label of a corresponding region in the labeled image information to each point in the point cloud information from different viewpoints based on position information of the point. The point cloud integration unit aligns the labeled point cloud information using labels common to the multiple labeled point cloud information.

[0004] Japanese Patent Application Laid-Open No. 2022-157660

[0005] The point cloud information processing device described in Patent Document 1 uses image information captured from different viewpoints to assign label information to a point cloud, which poses a problem that it is difficult to assign label information to a point cloud.

[0006] The information processing device of the present disclosure includes: a first acquisition means for generating a two-dimensional point cloud by converting a three-dimensional point cloud based on radar information; a second acquisition means for acquiring, based on an image captured using light, a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to objects included in the two-dimensional captured image; an assignment means for assigning the label information to the two-dimensional point cloud based on a positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image; and an output means for outputting output information in which the label information has been assigned to the point cloud based on the radar information.

[0007] The information processing system of the present disclosure comprises a mobile body that moves to generate the radar information obtained by scanning a scanning area using a radar, an imaging device that images the scanning area and generates the captured image, and the information processing device, wherein the mobile body includes a transmitting means that transmits the radar to the scanning area, a receiving means that receives reflected waves of the transmitted radar and generates the radar information related to the reflected waves, and a transmitting means that transmits the generated radar information.

[0008] The information processing method disclosed herein includes one or more computers: generating a two-dimensional point cloud by converting a three-dimensional point cloud based on radar information; acquiring a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to objects included in the two-dimensional captured image based on an image captured using light; assigning the label information to the two-dimensional point cloud based on the positional relationship between the two-dimensional point cloud and the objects in the two-dimensional captured image; and outputting output information in which the label information has been assigned to the point cloud based on the radar information.

[0009] The recording medium in the present disclosure has recorded thereon a program for causing one or more computers to execute the following operations: generate a two-dimensional point cloud by converting a three-dimensional point cloud based on radar information; obtain a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to objects included in the two-dimensional captured image based on an image captured using light; assign the label information to the two-dimensional point cloud based on the positional relationship between the two-dimensional point cloud and the objects in the two-dimensional captured image; and output information in which the label information is assigned to the point cloud based on the radar information.

[0010] According to the present disclosure, it becomes possible to easily assign label information to a point cloud.

[0011] 1 is a block diagram showing an example configuration of a first information processing system according to the present disclosure; FIG. 2 is a block diagram showing an example configuration of a first information processing device according to the present disclosure; FIG. 3 is a flowchart showing an example processing operation of the first information processing device according to the present disclosure; FIG. 4 is a diagram showing an example of an image including a two-dimensional point cloud obtained by projecting (a) a three-dimensional point cloud onto a two-dimensional point cloud on a projection plane and (b) adding a frame surrounding an object; FIG. 5 is a diagram showing an example of a two-dimensional captured image added with a frame surrounding an object; FIG. 6 is a diagram showing an example physical configuration of a first information processing device according to the present disclosure; FIG. 7 is a block diagram showing an example configuration of a first acquisition unit according to the present disclosure; FIG. 8 is a flowchart showing an example processing operation of the first acquisition unit according to the present disclosure; FIG. 9 is a block diagram showing an example configuration of a second acquisition unit according to the present disclosure; FIG. 10 is a flowchart showing an example processing operation of the second acquisition unit according to the present disclosure.

[0012] Hereinafter, in this disclosure, the drawings relate to one or more embodiments. In addition, in all drawings, similar components are given similar reference numerals and descriptions thereof will be omitted as appropriate.

[0013] First Embodiment (Overview) As shown in FIG. 1, an information processing system 100 includes a moving object 110, an image capturing device 120, and an information processing device 130.

[0014] The mobile object 110 moves to generate radar information by scanning a scanning area using a radar. The mobile object 110 includes a transmitting unit 111, a receiving unit 112, and a transmitting unit 113.

[0015] The transmitter 111 transmits radar waves to a scanning area. The receiver 112 receives reflected waves from the transmitted radar waves and generates radar information related to the reflected waves. The transmitter 113 transmits the generated radar information.

[0016] The imaging device 120 is a device for capturing an image of a scanning region and generating a captured image.

[0017] As shown in FIG. 2, the information processing device 130 includes a first acquisition unit 131, a second acquisition unit 132, an assignment unit 133, and an output unit 134.

[0018] The first acquisition unit 131 generates a two-dimensional point cloud by converting the three-dimensional point cloud based on the radar information.

[0019] The second acquisition unit 132 acquires, based on a captured image captured using light, a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud, and label information assigned to objects included in the two-dimensional captured image.

[0020] The assigning unit 133 assigns label information to the two-dimensional point cloud based on the positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image.

[0021] The output unit 134 outputs output information in which label information is added to the point cloud based on the radar information.

[0022] According to the information processing system 100, it is possible to assign label information to a point cloud using a captured image. At least one captured image is sufficient. Therefore, it is possible to easily assign label information to a point cloud.

[0023] Furthermore, with this information processing device 130, it is possible to assign label information to a point cloud using a captured image. At least one captured image is sufficient. Therefore, it is possible to easily assign label information to a point cloud.

[0024] The information processing device 130 executes information processing as shown in FIG.

[0025] The first acquisition unit 131 generates a two-dimensional point cloud by converting a three-dimensional point cloud based on radar information (step S101).

[0026] The second acquisition unit 132 acquires, based on the captured image captured using light, a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud, and label information assigned to objects contained in the two-dimensional captured image (step S102).

[0027] The assigning unit 133 assigns label information to the two-dimensional point cloud based on the positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image (step S103).

[0028] The output unit 134 outputs output information in which label information is added to the point cloud based on the radar information (step S104).

[0029] According to this information processing, it is possible to assign label information to a point cloud using a captured image. At least one captured image is sufficient. Therefore, it is possible to easily assign label information to a point cloud.

[0030] (Detailed Example) A detailed example of the information processing system 100 etc. will be described below.

[0031] (Configuration Example of Mobile Body 110) The mobile body 110 is an air vehicle such as a drone that moves by remote control or operation by an operator, or automatically according to a predetermined algorithm, etc. The mobile body 110 may be equipped with, for example, devices, apparatuses, etc. that realize the functions of a transmitter 111, a receiver 112, a transmitter 113, etc.

[0032] The moving body 110 is not limited to an aircraft, but may be a vehicle such as an automobile. The moving body 110 may further include a movement control unit (not shown) for controlling the movement of the moving body 110.

[0033] For radar, millimeter waves, which are radio waves with wavelengths of 1 to 10 mm (millimeters), are preferably used. One characteristic of millimeter waves is that they penetrate materials better than light. "Light" includes visible light and infrared light, and this also applies hereinafter.

[0034] Note that radar may use radio waves of various wavelengths, not just millimeter waves. For example, microwaves may be used, or microwaves with wavelengths longer than light may be used. Microwaves are radio waves with wavelengths of 1 meter or less, such as ultra-short waves, centimeter waves, millimeter waves, and submillimeter waves. Ultra-short waves, centimeter waves, and submillimeter waves have wavelengths of 0.1 to 1 m (meters), 1 to 10 cm (centimeters), and 0.1 to 1 mm, respectively.

[0035] The scan area is a predetermined area to be scanned by the radar, and may be, for example, an outdoor area or an indoor area.

[0036] In more detail, for example, the scanning area may be an area where there is a possibility that a metal object or the like is placed on the ground surface or is partially or completely buried underground. The metal object or the like can be detected using radar information obtained by scanning such a scanning area with a radar.

[0037] Furthermore, for example, the scanning area may be a predetermined area of ​​a building, etc. By using radar information obtained by scanning such a scanning area with a radar, it is possible to detect the condition of pipes, reinforcing bars, etc. arranged inside or outside the walls of the building, etc., and to discover abnormalities in the pipes, reinforcing bars, etc. Examples of buildings include, but are not limited to, buildings, bridges, etc.

[0038] The mobile object 110 moves, for example, with the transmitter 111 transmitting radar waves and the receiver 112 receiving the reflected waves. This allows the radar to scan a scanning area and acquire radar information related to the reflected waves. That is, the radar information is information acquired using the mobile object 110, and more specifically, information acquired using an aircraft such as a drone, a vehicle, or the like.

[0039] For example, if the mobile object 110 is a drone, it may transmit radar waves and receive their reflected waves while flying at a height of about 10 m. Various common radar transmission methods may be used. Examples of radar transmission methods include frequency-continuous modulation (FMCW), pulse, continuous wave Doppler (CWD), two-frequency CW, and pulse compression.

[0040] When the receiver 112 generates radar information related to the received reflected waves, the transmitter 113 transmits the radar information to the information processing device 130 via, for example, the network NT1. The network NT1 is typically a wireless line, but may include at least a wired line. The transmitter 113 may transmit the radar information in real time, or may collectively transmit multiple pieces of radar information generated at different times.

[0041] The radar information is, for example, information about a reflected wave of a radar transmitted into a scanning space. In detail, the radar information associates one or more of the intensity, observation position, observation direction, observation time, etc. of the reflected wave.

[0042] The observation position is a position in real space where observation is performed, and may be at least one of, for example, the radar transmission position, the reflected wave reception position, a position obtained using the transmission position and reception position, such as a position midway between the transmission position and reception position, etc. The observation position is expressed, for example, by latitude, longitude, height, etc., and may be obtained by providing the mobile body 110 with a GPS (Global Positioning System) function. Note that the observation position is not limited to the example given here.

[0043] The observation direction may be at least one of the radar transmission direction, the reflected wave reception direction, a direction obtained using the transmission direction and the reception direction, etc. The receiving unit 112 may include one or more antennas to obtain the reception direction.

[0044] The observation time is information indicating the time of observation, such as the observation time. The observation time may be at least one of the radar transmission time (e.g., the transmission time), the reception time of the reflected wave (e.g., the transmission time), a time associated with the transmission time and the reception time, such as midway between the transmission time and the reception time, etc. The observation time may be acquired by providing the mobile object 110 with a timekeeping function.

[0045] (Configuration example of the image capturing device 120) The image capturing device 120 is a device for capturing an image of the operation area using light. As described above, the light includes visible light and infrared light. For example, when visible light is used, the image capturing device 120 is a visible light camera. For example, when infrared light, near-infrared light, or far-infrared light is used, the image capturing device 120 is an infrared camera, a near-infrared camera, or a far-infrared camera, respectively.

[0046] The photographing device 120 generates a photographed image by photographing the scanning area, and transmits the photographed image to the information processing device 130 via the network NT2, for example.

[0047] The captured image is, for example, a color image such as an RGB image, but may also be a monochrome image.

[0048] The network NT2 is typically a wireless network, but may include at least a wired network. The networks NT1 and NT2 may be partly or entirely a common network, or partly or entirely different networks.

[0049] The image capturing device 120 may be fixed in position, or may be mounted on the above-described mobile body 110 or a mobile body different from the mobile body 110. This mobile body may be an aerial vehicle such as a drone, or a vehicle such as an automobile. In addition, this mobile body may be remotely controlled or operated by an operator, or may move automatically according to a predetermined algorithm or the like.

[0050] When the imaging device 120 is mounted on a moving body, the imaging device 120 may capture images of the scanning area while moving.

[0051] For example, when the image capturing device 120 is mounted on a moving body, the image capturing device 120 may capture images of the scanning area while moving. In this case, the image capturing device 120 may transmit captured images in real time, or may transmit multiple images captured at different times together.

[0052] The photographing device 120 may transmit photographing information in which at least one of the photographing location and the photographing time is associated with the photographed image.

[0053] The shooting position is the position in real space where the image was captured. The shooting position is expressed, for example, by latitude, longitude, and altitude, and may be acquired by providing the image capturing device 120 or a mobile object equipped with the same with a GPS (Global Positioning System) function. Note that the shooting position is not limited to the example given here.

[0054] The photographing period is the period when the photograph was taken, for example, the time of the photograph. Note that the photographing period is not limited to the example given here.

[0055] (Example of functional configuration of information processing device 130) (Regarding first acquisition unit 131) The first acquisition unit 131 generates a three-dimensional point cloud based on, for example, radar information transmitted from the transmission unit 113, and generates a two-dimensional point cloud by converting the three-dimensional point cloud.

[0056] In detail, for example, the first acquisition unit 131 generates three-dimensional information relating to the three-dimensional point cloud based on the radar information transmitted from the transmission unit 113 .

[0057] The three-dimensional point cloud is a group of points in three-dimensional space corresponding to the scanned area. For example, as described above, the scanned area may be inside an object, such as the inside of a wall of a building, or may be underground. Therefore, the three-dimensional point cloud may include a group of points representing the inside of an object, such as the inside of a wall of a building, or the underground.

[0058] For example, the three-dimensional information is information in which a three-dimensional point cloud and a certainty factor are associated with each other. The certainty factor is a value corresponding to the likelihood that an object exists in the associated three-dimensional point cloud. The certainty factor may be calculated based on the intensity of the reflected wave in the three-dimensional point cloud, and is a value indicating, for example, the probability that an object exists in the three-dimensional point cloud.

[0059] In more detail, for example, the 3D point cloud may be represented by identification information for identifying each of the 3D point clouds and the position of each of the 3D point clouds. In this case, the 3D information is information in which the identification information for identifying each of the 3D point clouds, the position of each of the 3D point clouds, and the reliability are associated with each other.

[0060] For example, the first acquisition unit 131 may calculate the distance at each point in three-dimensional space using a fast Fourier transform (FFT) based on the radar information, and then calculate the angle or position by integrating the calculated distances. Furthermore, for example, the first acquisition unit 131 may calculate a reliability for each point in three-dimensional space based on the radar information, the greater the reliability value becomes as the presence of an object at that point increases. This allows the first acquisition unit 131 to generate three-dimensional information.

[0061] Note that there may be multiple pieces of radar information. In this case, the multiple pieces of radar information may be generated and transmitted by each of the multiple mobile objects 110. Furthermore, the mobile object 110 may include multiple pairs of transmitters 111 and receivers 112. The multiple pieces of radar information may be generated by each of the multiple pairs of transmitters 111 and receivers 112 and transmitted from one or multiple transmitters 113.

[0062] The first acquisition unit 131 generates two-dimensional information related to the two-dimensional point cloud, for example, by converting the three-dimensional point cloud.

[0063] This transformation is a projection onto a predetermined projection plane. That is, the 2D point cloud is a point cloud obtained by projecting the 3D point cloud onto the projection plane. Figure 4(a) is a diagram showing an example of projecting the 3D point cloud onto the 2D point cloud on the projection plane.

[0064] The projection plane is a plane corresponding to the ground surface included in the scanning area or a plane parallel to the ground surface at a predetermined distance above or below the ground surface.

[0065] Such a conversion from a three-dimensional point cloud to a two-dimensional point cloud may be performed using a general technique for transforming an image, such as affine transformation, homography transformation, etc. Note that the conversion from a three-dimensional point cloud to a two-dimensional point cloud is not limited to a general technique for transforming an image, and may also use, for example, a technique for transforming a coordinate system.

[0066] The projection plane is not limited to the one exemplified here, and may be defined, for example, not parallel to the earth's surface, and may be defined using altitude or the like instead of distance to the earth's surface.

[0067] For example, the two-dimensional information is information for associating, for each two-dimensional point group, at least part of the following: identification information for identifying each two-dimensional point group, its position on the projection plane, the corresponding three-dimensional point group, and a certainty factor, which is a value corresponding to the likelihood that an object exists.

[0068] The corresponding three-dimensional point group is information about the three-dimensional point group of the projection source.

[0069] For example, the corresponding three-dimensional point clouds may be identification information for identifying each of the three-dimensional point clouds, and the identification information and the three-dimensional information may be used to associate each of the two-dimensional point clouds with each piece of information included in the three-dimensional information.

[0070] Furthermore, for example, the corresponding three-dimensional point cloud may include one or more of the information included in the three-dimensional information, i.e., identification information for identifying each of the three-dimensional point clouds, the position of each of the three-dimensional point clouds, and reliability. In this way, the two-dimensional information directly includes the information of the corresponding three-dimensional point cloud, thereby making it possible to associate information about the three-dimensional point cloud from which the two-dimensional information is projected.

[0071] The confidence level included in the two-dimensional information may be set based on the confidence level assigned to the corresponding three-dimensional point cloud, and may be the same as the confidence level assigned to the corresponding three-dimensional point cloud, for example.

[0072] However, when projecting onto a projection plane, a plurality of three-dimensional point groups may be projected onto a common position on the projection plane.

[0073] In such cases, the corresponding 3D point cloud may be the plurality of 3D point clouds, or may be a 3D point cloud that is representative of the plurality of 3D point clouds (e.g., the 3D point cloud that is closest to the projection plane, the 3D point cloud that is associated with the greatest confidence, etc.).

[0074] Furthermore, if the corresponding three-dimensional point cloud is a plurality of three-dimensional point clouds, the certainty factor included in the two-dimensional information may be, for example, an average value of the certainty factors associated with the plurality of three-dimensional point clouds. If the corresponding three-dimensional point cloud is a three-dimensional point cloud closest to the projection plane, the certainty factor included in the two-dimensional information may be, for example, the certainty factor associated with the closest three-dimensional point cloud. If the corresponding three-dimensional point cloud is a three-dimensional point cloud with the greatest certainty factor associated therewith, the certainty factor included in the two-dimensional information may be, for example, the certainty factor of the three-dimensional point cloud.

[0075] (Regarding the second acquisition unit 132) The second acquisition unit 132 acquires a two-dimensional captured image and label information assigned to an object included in the two-dimensional captured image based on the captured image of the scanning area transmitted from the imaging device 120.

[0076] A two-dimensional captured image is an image obtained by deforming a captured image of a scanning area so that the coordinate system used for the captured image of the scanning area is a common coordinate system for the two-dimensional point cloud. In other words, a two-dimensional captured image can also be said to be a captured image of a scanning area expressed in a common coordinate system for the two-dimensional point cloud. Note that although the captured image may also be a two-dimensional image, in this disclosure, in order to distinguish between images before and after deformation, the image before deformation is referred to as a captured image and the image after deformation is referred to as a two-dimensional captured image.

[0077] The target object is a predetermined object, for example, an object that may be present in the scanning area.

[0078] The label information may include at least one of class identification information and class reliability.

[0079] The class identification information is information for identifying the class (type of object) to which the object belongs. The class (type of object) may be one or more of metal objects, plants, rocks, etc.

[0080] The class identification information may be represented by, for example, one or a combination of letters, numbers, symbols, etc., that are assigned in advance to each class.

[0081] It should be noted that the classes and the methods of expressing them are not limited to those exemplified here.

[0082] The class reliability is a value indicating the likelihood that an object belongs to a class, and is expressed, for example, as a continuous value within a predetermined range.

[0083] Such label information may be obtained by human input or automatically using an object detection model.

[0084] The object detection model is a machine learning model for detecting objects and assigning class information to the objects, and when an image is input, the model outputs class information associated with the objects included in the image. The object detection model is preferably trained using training images and ground truth data including the class information of the objects in the images.

[0085] Such an object detection model may be a general machine learning model for detecting objects in images, and examples of techniques applied to such object detection models include R-CNN, YOLO, SSD, Fast R-CNN, and Faster R-CNN.

[0086] For example, if an object is buried underground, label information for the object may be assigned by placing a marker on the ground indicating the object's position, and by human input or automatic detection using an object detection model.

[0087] The label information may be associated with the position of a frame (for example, a rectangular frame circumscribing the object) surrounding the object in the two-dimensional captured image. FIG. 5 is a diagram showing an example of a two-dimensional captured image to which a frame surrounding the object has been added. In FIG. 5, the area corresponding to the object (image of second areas Qa to Qc) in the two-dimensional captured image is indicated by hatching. In this case, the label information may further include the position of the frame surrounding the object in the two-dimensional captured image. Note that the frame is not limited to a rectangle, and the shape, etc., may be changed as appropriate.

[0088] When label information is assigned to a two-dimensional captured image, the label information may be associated with each pixel corresponding to an object. The label information may be associated with the position of a frame surrounding the object in the two-dimensional captured image (e.g., a rectangular frame circumscribing the object). The label information may be associated with a region (mask) corresponding to the object in the two-dimensional captured image.

[0089] (Assignment Unit 133) The assignment unit 133 assigns label information to the two-dimensional point cloud based on the positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image.

[0090] (Assigning Label Information to a 2D Point Cloud) As described above, the 2D point cloud and the 2D captured image share a common coordinate system. Therefore, for example, the assigning unit 133 may assign label information to a 2D point cloud included in a region of an object in the 2D captured image, assuming that the 2D point cloud corresponds to the object.

[0091] When label information is assigned to a two-dimensional point cloud, the label information may be associated with each of the two-dimensional point clouds. The label information may be associated with the position of a frame surrounding an object in the two-dimensional point cloud (e.g., a rectangular frame circumscribing the object). In FIG. 4B, the point cloud corresponding to the object (the point cloud of the first regions Pa to Pc) is shown as a set of points. In this case, the label information may further include the position of the frame surrounding the object in the two-dimensional point cloud. Note that the frame is not limited to a rectangle, and the shape, etc., may be changed as appropriate. Furthermore, the label information is not limited to a frame surrounding the object, and may be associated with at least one of the two-dimensional point clouds in an appropriate manner, such as a region (mask) corresponding to the object in the two-dimensional point cloud.

[0092] At this time, when the certainty in the two-dimensional information is equal to or greater than a predetermined reference value, the assigning unit 133 may assign label information to the two-dimensional point cloud associated with the certainty. That is, the label information may be assigned to the two-dimensional point cloud whose certainty is equal to or greater than a predetermined reference value.

[0093] (Assigning Label Information to a 3D Point Cloud) As described above, the 2D information associates a 2D point cloud with a 3D point cloud from which the 2D point cloud is projected. Therefore, the assigning unit 133 may assign the label information to a 3D point cloud corresponding to the 2D point cloud to which the label information has been assigned, using the 2D point cloud to which the label information has been assigned and the 2D information. This allows the label information to be assigned to the 3D point cloud. In other words, the label information may be assigned to the 2D point cloud based on the positional relationship, and may be assigned to the 3D point cloud associated with the 2D point cloud by the 2D information.

[0094] When label information is assigned to a 3D point cloud, the label information may be associated with each of the 3D point clouds. The label information may be associated with the position of a frame surrounding an object in the 3D point cloud (e.g., a rectangular frame circumscribing the object). The label information may be associated with a region (mask) in the 3D point cloud corresponding to the object.

[0095] (Assignment of label information to point cloud within specific region) Furthermore, the assigning unit 133 may assign label information to a point cloud within a specific region different from the two-dimensional point cloud and the three-dimensional point cloud, based on the two-dimensional point cloud to which label information has been assigned. That is, label information may be further assigned to a point cloud within a specified specific region based on the two-dimensional point cloud to which label information has been assigned.

[0096] The specific region is an area that is appropriately specified. For example, a specific plane different from the projection plane may be specified, and the specific region may be determined based on the specific plane.

[0097] The specific plane may be specified using, for example, a distance (e.g., height) from the ground surface. The specific region is, for example, a region near the specific plane. In detail, for example, the specific region is a region within a predetermined range in the vertical direction from the specific plane. This predetermined range may be a height specified by the user or may be determined in advance.

[0098] When label information is assigned to a point cloud of a specific region, the label information may be associated with each point cloud of the specific region. The label information may be associated with the position of a frame surrounding an object (e.g., a rectangular frame circumscribing the object) in the point cloud of the specific region. The label information may be associated with a region (mask) corresponding to the object in the point cloud of the specific region.

[0099] Here, for example, there may be a plurality of 2D point clouds to which label information has been assigned among the 2D point clouds corresponding to the point cloud of a specific region. In this case, the label information may be preferentially selected in descending order of reliability associated with the 2D point cloud, and assigned to the point cloud of the specific region corresponding to the selected 2D point cloud.

[0100] (Regarding the Output Unit 134) The output unit 134 outputs output information in which label information is added to a point cloud based on radar information. This point cloud may be one or more of a two-dimensional point cloud, a three-dimensional point cloud, and a point cloud within a specific region.

[0101] That is, the output information may be information in which label information is added to a two-dimensional point cloud, information in which label information is added to a three-dimensional point cloud, or information in which label information is added to a point cloud within a specific region.

[0102] The destination of the output information may be one or more of another device connected via a network, a storage unit (not shown), a display unit (not shown), etc. The storage unit and the display unit may each be provided in another device.

[0103] (Example of Physical Configuration of Information Processing Device 130) As shown in FIG. 6, the information processing device 130 physically includes, for example, a bus 1010, a processor 1020, a memory 1030, a storage device 1040, a network interface 1050, an input interface 1060, and an output interface 1070.

[0104] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, network interface 1050, input interface 1060, and output interface 1070. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0105] The processor 1020 is implemented as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0106] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0107] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules for realizing the functions of the information processing device 130 that includes the storage device 1040. The processor 1020 reads each of these program modules into the memory 1030 and executes them to realize the function corresponding to the program module.

[0108] The network interface 1050 is an interface for connecting the information processing device 130 having the network interface 1050 to the network NT.

[0109] The input interface 1060 is an interface for the user to input information, and is configured from, for example, a touch panel, a keyboard, a mouse, and the like.

[0110] The output interface 1070 is an interface for presenting information to the user, and is configured, for example, by a liquid crystal panel, an organic EL (Electro-Luminescence) panel, or the like.

[0111] Note that the physical configuration of the information processing device 130 is not limited to this. For example, the information processing device 130 may be composed of multiple devices. In this case, each device may be, for example, a computer or the like having a physical configuration similar to that of the information processing device 130 shown in FIG. 6.

[0112] (Operations and Effects) As described above, according to this embodiment, the information processing device 130 includes the first acquisition unit 131, the second acquisition unit 132, the assignment unit 133, and the output unit 134.

[0113] The first acquisition unit 131 generates a two-dimensional point cloud by converting the three-dimensional point cloud based on the radar information. The second acquisition unit 132 acquires a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to objects included in the two-dimensional captured image based on an image captured using light. The assignment unit 133 assigns label information to the two-dimensional point cloud based on the positional relationship between the two-dimensional point cloud and the objects in the two-dimensional captured image. The output unit 134 outputs output information in which the label information is assigned to the point cloud based on the radar information.

[0114] This allows label information to be assigned to a point cloud using the captured image. At least one captured image is sufficient. Therefore, label information can be easily assigned to a point cloud.

[0115] According to this embodiment, the two-dimensional point cloud is a point cloud obtained by projecting a three-dimensional point cloud onto a projection plane.

[0116] This makes it possible to easily generate a two-dimensional point cloud to which the label information assigned to the captured image can be applied, thereby making it possible to easily assign label information to the point cloud.

[0117] According to this embodiment, label information is further assigned to the three-dimensional point cloud, and the output information is information in which the label information is assigned to the three-dimensional point cloud.

[0118] This makes it possible to output output information in which label information is assigned to the 3D point cloud. Such output information can be used, for example, as training data when training a machine learning model that uses the 3D point cloud as input and detects objects included in the 3D point cloud. Generally, it is often difficult for a person to identify the location of an object in a 3D point cloud, making it difficult to create such training data. Using output information in which label information is assigned to the 3D point cloud makes it possible to easily train a machine learning model that detects objects included in the 3D point cloud.

[0119] According to this embodiment, label information is further assigned to the point cloud within the specified specific region based on the two-dimensional point cloud to which the label information has been assigned, and the output information is information in which the label information has been assigned to the point cloud within the specific region.

[0120] This allows point cloud label information to be assigned to any region, making it possible to easily assign label information to point clouds within any region.

[0121] According to this embodiment, the label information includes at least one of class identification information for identifying the class to which the object belongs, and class confidence indicating the likelihood that the object belongs to the class.

[0122] This allows the output information to be used as learning data and to be easily viewed on the display unit, making it possible to utilize the output information for a variety of purposes.

[0123] According to this embodiment, by generating a two-dimensional point cloud, two-dimensional information is generated for each of the two-dimensional point clouds to associate the position on the projection plane, the corresponding three-dimensional point cloud, and a certainty factor, which is a value corresponding to the likelihood that an object exists. Label information is assigned to the two-dimensional point cloud based on the positional relationship, and is assigned to the three-dimensional point cloud associated with the two-dimensional point cloud in the two-dimensional information.

[0124] This allows label information to be added to the 3D point cloud, making it possible to use the output information for various purposes.

[0125] According to this embodiment, label information is assigned to a two-dimensional point cloud whose certainty is equal to or greater than a predetermined reference value.

[0126] This reduces the number of times that label information is assigned to a 2D point cloud with a low degree of certainty, which is usually a point cloud where it is unclear whether an object exists or not, and therefore improves the accuracy of assigning label information.

[0127] According to this embodiment, generating a two-dimensional point cloud further generates three-dimensional information based on radar information, in which a three-dimensional point cloud is associated with a certainty factor, which is a value corresponding to the likelihood that an object exists in the three-dimensional point cloud. When multiple three-dimensional point clouds are projected onto a common position on a projection plane, the certainty factor for the two-dimensional point cloud obtained by projecting the multiple three-dimensional point clouds onto the projection plane is one of the following (1) to (3): (1) The certainty factor of the three-dimensional point cloud that is closest to the projection plane among the multiple three-dimensional point clouds. (2) The average value of the certainty factors of the multiple three-dimensional point clouds. (3) The maximum value of the certainty factors of the multiple three-dimensional point clouds.

[0128] This allows an appropriate degree of certainty to be easily associated with a two-dimensional point cloud, and label information to be assigned to the two-dimensional point cloud using this degree of certainty, thereby enabling label information to be assigned to the two-dimensional point cloud with high accuracy and ease.

[0129] Second Embodiment In this embodiment, a detailed example of the first acquisition unit 131 and the first process executed by the first acquisition unit 131 will be described.

[0130] As shown in FIG. 7, the first acquisition unit 131 includes a radar information acquisition unit 131a, a three-dimensional information acquisition unit 131b, and a projection unit 131c.

[0131] The radar information acquisition unit 131a acquires radar information.

[0132] The three-dimensional information acquisition unit 131b acquires three-dimensional information relating to the three-dimensional point cloud based on the radar information acquired by the radar information acquisition unit 131a.

[0133] The projection unit 131c uses the three-dimensional information acquired by the three-dimensional information acquisition unit 131b to generate two-dimensional information related to a two-dimensional point group by projecting the three-dimensional point group onto a projection plane.

[0134] The first acquisition unit 131 executes a first acquisition process (step S101) as shown in FIG. 8, for example.

[0135] The radar information acquisition unit 131a acquires radar information from the transmission unit 113 via the network NT1, for example (step S101a).

[0136] The three-dimensional information acquisition unit 131b acquires three-dimensional information relating to the three-dimensional point cloud based on the radar information acquired in step S101a (step S101b).

[0137] The projection unit 131c uses the three-dimensional information acquired in step S101b to generate two-dimensional information about a two-dimensional point group obtained by projecting the three-dimensional point group onto a projection plane (step S101c), and returns to information processing (see Figure 3).

[0138] For example, when the projection unit 131c receives information for identifying a projection plane, it projects the three-dimensional point cloud onto the projection plane to generate a two-dimensional point cloud on the projection plane. The projection unit 131c assigns identification information to each of the two-dimensional point clouds according to a predetermined rule. The projection unit 131c associates this identification information with the position of the two-dimensional point cloud on the projection plane. The projection unit 131c also calculates a certainty factor for the two-dimensional point cloud based on the certainty factor associated with the three-dimensional point cloud corresponding to the two-dimensional point cloud. The projection unit 131c then generates two-dimensional information in which each of the two-dimensional point clouds is associated with the identification information, the position of the two-dimensional point cloud on the projection plane, the corresponding three-dimensional point cloud, and a certainty factor, which is a value corresponding to the likelihood that an object exists in the two-dimensional point cloud.

[0139] (Operations and Effects) As described above, according to this embodiment, the first acquisition unit 131 includes the radar information acquisition unit 131a, the three-dimensional information acquisition unit 131b, and the projection unit 131c.

[0140] The radar information acquisition unit 131a acquires radar information. The three-dimensional information acquisition unit 131b acquires three-dimensional information about a three-dimensional point cloud based on the radar information acquired by the radar information acquisition unit 131a. The projection unit 131c uses the three-dimensional information acquired by the three-dimensional information acquisition unit 131b to generate two-dimensional information about a two-dimensional point cloud by projecting the three-dimensional point cloud onto a projection plane.

[0141] This makes it possible to generate two-dimensional information relating to a two-dimensional point group obtained by projecting a three-dimensional point group onto a projection plane, thereby making it possible to easily generate two-dimensional information relating to a two-dimensional point group.

[0142] Third Embodiment In this embodiment, a detailed example of the second acquisition unit 132 and the second process executed by the second acquisition unit 132 will be described.

[0143] As shown in FIG. 9, the second acquisition unit 132 includes a captured image acquisition unit 132a, an image transformation unit 132b, and an image label acquisition unit 132c.

[0144] The photographed image acquisition unit 132a acquires a photographed image captured using light.

[0145] The image transformation unit 132b transforms the captured image acquired by the captured image acquisition unit 132a into a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud.

[0146] The image label acquisition unit 132c acquires label information assigned to the object included in the two-dimensional captured image acquired by the image deformation unit 132b.

[0147] The second acquisition unit 132 executes, for example, a second acquisition process (step S102) as shown in FIG.

[0148] The photographed image acquisition unit 132a acquires a photographed image from the photographing device 120 via, for example, the network NT2 (step S102a).

[0149] The image transformation unit 132b transforms the captured image acquired in step S101a into a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud (step S102b).

[0150] For example, the image deformation unit 132b deforms the captured image into a two-dimensional captured image so that the positions in real space of pixels included in the captured image coincide with the positions in real space of the two-dimensional point cloud.

[0151] The process of step S102b may be performed as needed. For example, the process of step S102b may be performed when the coordinate systems of the captured image acquired in step S101a and the two-dimensional point cloud generated in step S101c are different, but may not be performed when they are the same.

[0152] The image label acquisition unit 132c acquires the label information assigned to the object included in the two-dimensional captured image acquired in step S102b (step S102c), and the process returns to the information processing (see FIG. 3).

[0153] In step S102c, the label information may be acquired by human input, as described above, or may be acquired using an object detection model. Furthermore, since the 2D captured image is an image obtained by deforming the captured image, the objects included in the 2D captured image and the captured image are the same. Therefore, in step S102c, instead of acquiring label information assigned to objects included in the 2D captured image, label information assigned to objects included in the captured image may be acquired.

[0154] (Operations and Effects) As described above, according to this embodiment, the second acquisition unit 132 includes the captured image acquisition unit 132a, the image transformation unit 132b, and the image label acquisition unit 132c.

[0155] The captured image acquisition unit 132a acquires a captured image captured using light. The image transformation unit 132b transforms the acquired captured image into a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud. The image label acquisition unit 132c acquires label information assigned to objects included in the two-dimensional captured image.

[0156] This makes it possible to obtain label information attached to objects included in a two-dimensional captured image, thereby making it possible to easily obtain label information attached to objects.

[0157] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0158] In addition, although the flowcharts used in the above description show a sequence of steps (processes), the order of steps executed in each embodiment is not limited to the sequence shown in the flowcharts. In each embodiment, the order of steps shown in the diagrams can be changed as long as it does not cause any problems in terms of the content.

[0159] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0160] 1. An information processing device comprising: a first acquisition means for generating a two-dimensional point cloud by converting a three-dimensional point cloud based on radar information; a second acquisition means for acquiring, based on an image captured using light, a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to objects included in the two-dimensional captured image; an assignment means for assigning the label information to the two-dimensional point cloud based on a positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image; and an output means for outputting output information in which the label information has been assigned to a point cloud based on the radar information. 2. The information processing device described in 1., wherein the two-dimensional point cloud is a point cloud obtained by projecting the three-dimensional point cloud onto a projection plane. 3. The information processing device described in 1. or 2., wherein the label information is further assigned to the three-dimensional point cloud, and the output information is information in which the label information has been assigned to the three-dimensional point cloud. 4. The information processing device according to 1. or 2., wherein the label information is further assigned to a point cloud within a specified specific region based on the two-dimensional point cloud to which the label information has been assigned, and the output information is information in which the label information has been assigned to the point cloud within the specific region. 5. The information processing device according to any one of 1. to 4., wherein the label information includes at least one of class identification information for identifying a class to which the object belongs and a class confidence indicating the likelihood that the object belongs to the class. 6. The information processing device according to 2., wherein generating the two-dimensional point cloud generates two-dimensional information for associating, for each of the two-dimensional point clouds, a position on the projection plane, the corresponding three-dimensional point cloud, and a confidence level which is a value corresponding to the likelihood that an object exists, and the label information is assigned to the two-dimensional point cloud based on the positional relationship, and assigned to the three-dimensional point cloud associated with the two-dimensional point cloud in the two-dimensional information. 7. The information processing device according to 6., wherein the label information is assigned to the two-dimensional point cloud whose confidence level is equal to or greater than a predetermined reference value.8. The information processing device according to 2, 6, or 7, wherein generating the two-dimensional point cloud further generates three-dimensional information that associates a three-dimensional point cloud with a certainty, the certainty being a value corresponding to the likelihood that an object exists in the three-dimensional point cloud, based on the radar information, and when a plurality of the three-dimensional point clouds are projected onto a common position on the projection plane, the certainty for the two-dimensional point cloud obtained by projecting the plurality of three-dimensional point clouds onto the projection plane is any one of the certainty for a three-dimensional point cloud that is closest to the projection plane among the plurality of three-dimensional point clouds, the average value of the certainty for the plurality of three-dimensional point clouds, and the maximum value of the certainty for the plurality of three-dimensional point clouds. 9. The information processing device according to 2, wherein the first acquisition means includes: radar information acquisition means that acquires the radar information; three-dimensional information acquisition means that acquires three-dimensional information for the three-dimensional point cloud based on the acquired radar information; and projection means that uses the three-dimensional information to generate two-dimensional information for the two-dimensional point cloud obtained by projecting the three-dimensional point cloud onto the projection plane. the second acquisition means includes: captured image acquisition means for acquiring a captured image captured using the light, image transformation means for transforming the acquired captured image into a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud, and image label acquisition means for acquiring the label information assigned to the object included in the two-dimensional captured image. 11. An information processing system comprising: a mobile body that moves to generate the radar information obtained by scanning a scan area using a radar, an imaging device for imaging the scan area and generating the captured image, and the information processing device described in any one of 1. to 10., wherein the mobile body includes: transmission means for emitting the radar to the scan area, reception means for receiving reflected waves of the emitted radar and generating the radar information related to the reflected waves, and transmission means for transmitting the generated radar information.12. An information processing method in which one or more computers: generate a two-dimensional point cloud by converting a three-dimensional point cloud based on radar information; acquire a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to objects included in the two-dimensional captured image based on an image captured using light; assign the label information to the two-dimensional point cloud based on a positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image; and output information in which the label information is assigned to the point cloud based on the radar information. A program for causing one or more computers to execute the following steps: generate a two-dimensional point cloud by converting a three-dimensional point cloud based on radar information; acquire a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to objects included in the two-dimensional captured image based on an image captured using light; assign the label information to the two-dimensional point cloud based on the positional relationship between the two-dimensional point cloud and the objects in the two-dimensional captured image; and output information in which the label information has been assigned to the point cloud based on the radar information.

[0161] This application claims priority based on Japanese Patent Application No. 2023-129864, filed on August 9, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0162] REFERENCE SIGNS LIST 100 Information processing system 110 Mobile object 111 Transmission unit 112 Reception unit 113 Transmission unit 120 Imaging device 130 Information processing device 131 First acquisition unit 131a Radar information acquisition unit 131b Three-dimensional information acquisition unit 131c Projection unit 132 Second acquisition unit 132a Photographed image acquisition unit 132b Image deformation unit 132c Image label acquisition unit 133 Assignment unit 134 Output unit

Claims

1. A first acquisition means that generates a two-dimensional point cloud by converting a three-dimensional point cloud based on radar information, A second acquisition means for acquiring a two-dimensional captured image represented in a coordinate system common to the two-dimensional point cloud, and label information assigned to objects included in the two-dimensional captured image, based on a captured image taken using light. A means for assigning label information to the two-dimensional point cloud based on the positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image, The system includes output means that outputs output information in which the label information is assigned to a point cloud based on the radar information. Information processing device.

2. The aforementioned two-dimensional point cloud is a point cloud obtained by projecting the aforementioned three-dimensional point cloud onto the projection plane. The information processing apparatus according to claim 1.

3. The aforementioned label information is further assigned to the three-dimensional point cloud, The output information is information obtained by assigning the label information to the three-dimensional point cloud. The information processing apparatus according to claim 1 or 2.

4. The label information is further assigned to the point cloud within a specified specific region based on the two-dimensional point cloud to which the label information has been assigned. The output information is information obtained by assigning the label information to the point cloud within the specific region. The information processing apparatus according to claim 1 or 2.

5. The label information includes at least one of class identification information for identifying the class to which the object belongs, and a class confidence score indicating the likelihood that the object belongs to the class. The information processing apparatus according to claim 1 or 2.

6. In generating the aforementioned two-dimensional point cloud, two-dimensional information is generated to associate each of the two-dimensional point clouds with its position in the projection plane, the corresponding three-dimensional point cloud, and a confidence level, which is a value corresponding to the likelihood of an object existing. The label information is assigned to the two-dimensional point cloud based on the positional relationship, and is assigned to the three-dimensional point cloud associated with the two-dimensional point cloud using the two-dimensional information. The information processing apparatus according to claim 2.

7. In generating the aforementioned two-dimensional point cloud, three-dimensional information is further generated based on the radar information, in which the three-dimensional point cloud and a confidence level, which is a value corresponding to the likelihood that an object exists in the three-dimensional point cloud, are associated. When multiple three-dimensional point clouds are projected to a common position in the projection plane, the confidence level for the two-dimensional point cloud projected onto the projection plane is one of the following: the confidence level of the three-dimensional point cloud closest to the projection plane, the average confidence level of the multiple three-dimensional point clouds, or the maximum confidence level of the multiple three-dimensional point clouds. The information processing apparatus according to claim 2 or 6.

8. A mobile body that moves to generate radar information by scanning a scanning area using radar, A shooting device for capturing the aforementioned scanning area and generating an image, The information processing device comprises the information processing device described in claim 1 or 2, The aforementioned moving body is A transmitting means for transmitting the radar in the scanning area, Receiving means for receiving the reflected waves of the transmitted radar and generating radar information relating to the reflected waves, Includes a transmission means for transmitting the generated radar information. Information processing system.

9. One or more computers, Based on radar information, a 2D point cloud is generated by converting the 3D point cloud. Based on the captured image taken using light, a two-dimensional captured image represented in a coordinate system common to the two-dimensional point cloud, and label information assigned to the objects included in the two-dimensional captured image are obtained. Based on the positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image, the label information is assigned to the two-dimensional point cloud. Output information is generated by adding the label information to the point cloud based on the radar information. Information processing methods.

10. On one or more computers, Based on radar information, a 2D point cloud is generated by converting the 3D point cloud. Based on the captured image taken using light, a two-dimensional captured image represented in a coordinate system common to the two-dimensional point cloud, and label information assigned to the objects included in the two-dimensional captured image are obtained. Based on the positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image, the label information is assigned to the two-dimensional point cloud. A program for outputting output information in which the label information is added to a point cloud based on the radar information.