Information processing device, information processing method, and program

Abstract

This information processing device comprises a first acquisition unit, a second acquisition unit, an integration unit, and an identification unit. The first acquisition unit and the second acquisition unit acquire first information and second information, which are generated using respective electromagnetic waves of a first frequency and a second frequency different from each other, and in which positions in a target region are respectively associated with a first reliability and a second reliability indicating the likelihood that an object is present in said positions. On the basis of the first reliability and second reliability for respective sub-regions corresponding to the positions in the target region, the integration unit generates reliability map information indicating the reliability of each of the sub-regions. The identification unit identifies the region in which the object is present by processing the reliability map information.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to an information processing device, an information processing method, and a recording medium.

[0002] Patent Document 1 discloses an image signal processing method for detecting objects buried underground, which includes a three-dimensional data acquisition step, a moving average calculation step, an imaging step, a peak sharpness calculation step, and a buried object identification step.

[0003] The three-dimensional data acquisition step acquires the numerical values ​​of each element of the three-dimensional matrix data in a horizontal plane direction based on the ground surface and in a depth direction from the ground surface toward the ground. The moving average calculation step calculates a moving average in the horizontal plane based on the ground surface. The imaging step creates a three-dimensional image based on the three-dimensional data acquisition step and the moving average calculation step.

[0004] The peak sharpness calculation step calculates the peak sharpness from the cross-correlation between adjacent images, and the buried object identification step identifies the presence or absence of a buried object based on the condition that maximizes the peak sharpness.

[0005] According to Patent Document 1, the above three-dimensional matrix data (GPR measurement data) is obtained by processing detection signals from a ground-penetrating (GPR) radar.

[0006] Furthermore, according to Patent Document 1, when the target is a buried object such as a landmine, the value of the reflected wave from the landmine changes from positive to negative, causing the image of the buried object in the GPR measurement data to appear continuously in the depth direction, disappear once, and then reappear with the color inverted. In contrast, in the case of clutter C, which is unwanted reflection caused by heterogeneity such as gravel contained in the ground, the GPR measurement data changes finely and irregularly. For this reason, Patent Document 1 describes that the target and clutter C can be distinguished by calculating a moving average in a horizontal plane based on the ground surface.

[0007] Japanese Patent Application Laid-Open No. 2007-285781

[0008] However, clutter, which is unwanted reflection to be removed, is not limited to sand and gravel, but may also be caused by objects with strong reflection intensity, such as vegetation. With the technology described in Patent Document 1, it may be difficult to identify such clutter.

[0009] Therefore, the technology described in Patent Document 1 has a problem in that it is difficult to accurately identify an object from information generated using electromagnetic waves.

[0010] The information processing device of the present disclosure includes: a first acquisition means for acquiring first information, which is generated using electromagnetic waves of a first frequency, and which associates a position in a target area with a first reliability indicating the likelihood that an object exists at that position; a second acquisition means for acquiring second information, which is generated using electromagnetic waves of a second frequency different from the first frequency, and which associates a position in the target area with a second reliability indicating the likelihood that an object exists at that position; an integration means for generating reliability map information indicating the reliability of each of the sub-areas corresponding to each position in the target area, based on the first reliability and the second reliability for each of the sub-areas; and an identification means for identifying the area in which the object exists by processing the reliability map information.

[0011] The information processing method of the present disclosure includes one or more computers: acquiring first information generated using electromagnetic waves of a first frequency, associating a position in a target area with a first reliability indicating the likelihood that an object exists at that position; acquiring second information generated using electromagnetic waves of a second frequency different from the first frequency, associating a position in the target area with a second reliability indicating the likelihood that an object exists at that position; generating reliability map information indicating the reliability of each of the sub-areas corresponding to each position in the target area based on the first reliability and the second reliability for the sub-areas; and processing the reliability map information to identify the area where the object exists.

[0012] The recording medium of the present disclosure has recorded thereon a program for causing one or more computers to: acquire first information, which is generated using electromagnetic waves of a first frequency and associates a position in a target area with a first reliability indicating the likelihood that an object exists at that position; acquire second information, which is generated using electromagnetic waves of a second frequency different from the first frequency and associates a position in the target area with a second reliability indicating the likelihood that an object exists at that position; generate reliability map information indicating the reliability of each of the sub-areas corresponding to each position in the target area based on the first reliability and the second reliability for the sub-areas; and identify the area where the object exists by processing the reliability map information.

[0013] According to the present disclosure, it becomes possible to accurately identify an object from information generated using electromagnetic waves.

[0014] 1 is a block diagram illustrating a configuration example of a first information processing device according to the present disclosure. FIG. 2 is a flowchart illustrating an example of a processing operation of the first information processing device according to the present disclosure. FIG. 3 is a block diagram illustrating an example of a configuration of a first information processing system according to the present disclosure. FIG. 4 is a block diagram illustrating a detailed configuration example of the first information processing device according to the present disclosure. FIG. 5 is a flowchart illustrating a detailed processing example of the first information processing device according to the present disclosure. FIG. 6 is a block diagram illustrating an example of a configuration of a first acquisition unit according to the present disclosure. FIG. 7 is a flowchart illustrating an example of a processing operation of the first acquisition unit according to the present disclosure. FIG. 8 is a diagram illustrating an example of projecting a three-dimensional point cloud onto a two-dimensional point cloud on a projection plane. FIG. 9 is a diagram illustrating another example of first information. FIG. 10 is a block diagram illustrating an example of a configuration of a second acquisition unit according to the present disclosure. FIG. 11 is a flowchart illustrating an example of a processing operation of the second acquisition unit according to the present disclosure. FIG. 12 is a diagram illustrating an example of second information. FIG. 13 is a diagram for explaining integration processing performed by an integration unit. FIG. 14 is a diagram illustrating an example of reliability map information. FIG. 15 is a diagram illustrating an example of a physical configuration of a first information processing device according to the present disclosure. FIG. 16 is a block diagram illustrating an example of a configuration of an identification unit according to the present disclosure. FIG. 17 is a flowchart illustrating an example of a processing operation of the identification unit according to the present disclosure. FIG. 18 is a block diagram illustrating an example of a configuration of a second information processing device according to the present disclosure.

[0015] Hereinafter, in this disclosure, the drawings relate to one or more embodiments. In addition, in all drawings, similar components are given similar reference numerals and descriptions thereof will be omitted as appropriate.

[0016] First Embodiment (Overview) As shown in FIG. 1, an information processing device 100 includes a first acquisition unit 110, a second acquisition unit 120, an integration unit 130, and an identification unit 140.

[0017] The first acquisition unit 110 acquires first information generated using electromagnetic waves of a first frequency, which associates a position in a target area with a first reliability indicating the likelihood that an object exists at that position.

[0018] The second acquisition unit 120 acquires second information generated using electromagnetic waves of a second frequency different from the first frequency, and associating a position in the target area with a second reliability indicating the likelihood that an object exists at that position.

[0019] The integration unit 130 generates reliability map information indicating the reliability of each of the sub-regions based on the first reliability and the second reliability in each of the sub-regions corresponding to each position in the target region.

[0020] The identification unit 140 processes the reliability map information to identify the area where the object exists.

[0021] According to the information processing device 100, the reliability after integration is a value that can comprehensively consider the first reliability and the second reliability for each sub-region obtained using electromagnetic waves of different frequencies. Therefore, by using the reliability after integration to identify the region where the object exists, the object can be identified by comprehensively considering the reliabilities obtained using electromagnetic waves of different frequencies. Therefore, it is possible to accurately identify the object from information generated using electromagnetic waves.

[0022] The information processing device 100 executes information processing as shown in FIG.

[0023] The first acquisition unit 110 acquires first information generated using electromagnetic waves of a first frequency, which associates a position in a target area with a first reliability indicating the likelihood that an object exists at that position (step S110).

[0024] The second acquisition unit 120 acquires second information generated using electromagnetic waves of a second frequency different from the first frequency, and associates a position in the target area with a second reliability indicating the likelihood that an object exists at that position (step S120).

[0025] The integration unit 130 generates reliability map information indicating the reliability of each of the sub-regions based on the first reliability and the second reliability in each of the sub-regions corresponding to each position in the target region (step S130).

[0026] The identification unit 140 processes the reliability map information to identify the area where the object exists (step S140).

[0027] According to this information processing, the reliability after integration is a value that can comprehensively consider the first reliability and the second reliability for each sub-region obtained using electromagnetic waves of different frequencies. Therefore, by using the reliability after integration to identify the region where the object exists, the object can be identified by comprehensively considering the reliabilities obtained using electromagnetic waves of different frequencies. Therefore, it is possible to accurately identify the object from information generated using electromagnetic waves.

[0028] (Detailed Example) Hereinafter, a detailed example of the information processing device 100 etc. will be described.

[0029] (Regarding target area and target object) The target area is an area observed using electromagnetic waves of the first frequency and the second frequency, and may be predetermined. The target area may be, for example, a two-dimensional area (e.g., an area on a plane) or a three-dimensional area. Also, for example, the target area may be outdoors or indoors. The target area may include, for example, an area where a predetermined target object may exist.

[0030] In more detail, for example, the target area may include a predetermined area near the ground surface (i.e., a predetermined range from the ground surface). The predetermined area near the ground surface may be, for example, an area where an object such as a metal object may be placed on the ground surface or may be partially or completely buried underground. By setting such a target area, it is possible to detect an object such as a metal object.

[0031] Furthermore, for example, the target area may be a predetermined area of ​​a building or the like. By setting such a target area, it is possible to detect the state of pipes, reinforcing bars, etc., arranged inside or outside the walls of the building or the like, and to detect abnormalities in the target objects such as pipes, reinforcing bars, etc. Examples of buildings include, but are not limited to, buildings and bridges.

[0032] (Regarding Electromagnetic Waves) The first frequency and the second frequency may be different from each other. Generally, the lower the frequency of the electromagnetic wave, the higher the transparency, but the lower the resolution of point cloud information, images, and the like obtained using the electromagnetic wave.

[0033] The electromagnetic waves of the first frequency are, for example, electromagnetic waves of a frequency used in radar, and are millimeter waves, which are radio waves with a wavelength of 1 to 10 mm (millimeters). The electromagnetic waves of the second frequency are, for example, visible light and infrared light (hereinafter, these will be collectively referred to as "light"). A characteristic of millimeter waves is that they have better permeability through materials than light.

[0034] The electromagnetic waves of the second frequency may be millimeter waves. Furthermore, the electromagnetic waves of the first and second frequencies are not limited to millimeter waves and may be radio waves of various wavelengths, such as microwaves with wavelengths longer than light. Microwaves are radio waves with wavelengths of 1 meter or less, such as ultrashort waves, centimeter waves, millimeter waves, and submillimeter waves. Ultrashort waves, centimeter waves, and submillimeter waves have wavelengths of 0.1 to 1 m (meters), 1 to 10 cm (centimeters), and 0.1 to 1 mm, respectively.

[0035] (Configuration Example of Information Processing System S1) FIG. 3 shows a configuration example of the information processing system S1 including the information processing device 100.

[0036] As described above, the target region may be either a two-dimensional region or a three-dimensional region, but the following description will be given using an example in which the target region is a two-dimensional region. Also, the following description will be given using an example in which the electromagnetic wave of the first frequency is a millimeter wave and the electromagnetic wave of the second frequency is light.

[0037] The information processing system S1 includes a moving object 180, an image capturing device 190, and an information processing device 100.

[0038] (Regarding the Mobile Object 180) The mobile object 180 moves to generate first observation information obtained by observing a target area using a radar that irradiates electromagnetic waves of a first frequency.

[0039] The mobile object 180 is an air vehicle such as a drone that moves by remote control or operation by an operator or automatically according to a predetermined algorithm, etc. The mobile object 180 may be equipped with, for example, devices and apparatuses for realizing the functions of a transmitter 181, a receiver 182, a transmitter 183, etc., which will be described later.

[0040] The moving body 180 is not limited to an aircraft, but may be a vehicle such as an automobile. The moving body 180 may further include a movement control unit (not shown) for controlling the movement of the moving body 180.

[0041] The mobile unit 180 functionally includes, for example, a sending unit 181 , a receiving unit 182 , and a transmitting unit 183 .

[0042] The transmitter 181 transmits electromagnetic waves of a first frequency to a target area. The receiver 182 receives reflected waves of the transmitted electromagnetic waves of the first frequency and generates first observation information related to the reflected waves. The transmitter 183 transmits the generated first observation information.

[0043] For example, the mobile object 180 moves while the transmitter 181 transmits electromagnetic waves of a first frequency and the receiver 182 receives the reflected waves. This allows the target area to be scanned with the electromagnetic waves of the first frequency and first observation information related to the reflected waves to be acquired. That is, the first observation information is information acquired using the electromagnetic waves of the first frequency, and more specifically, is information indicating the results of observing the target area using, for example, an aircraft such as a drone, a vehicle, or the like. When the target area includes a predetermined area on the earth's surface, the first observation information is information indicating the results of observing the earth's surface using the electromagnetic waves of the first frequency.

[0044] For example, if the mobile object 180 is a drone, it may transmit electromagnetic waves of the first frequency and receive the reflected waves while flying at a height of about 10 m. Various common methods may be used to transmit the electromagnetic waves of the first frequency. Examples of methods for transmitting the electromagnetic waves of the first frequency include frequency-continuous modulation (FMCW), pulse, continuous wave Doppler (CWD), two-frequency CW, and pulse compression.

[0045] When the receiver 182 generates first observation information regarding the received reflected wave, the transmitter 183 transmits the first observation information to the information processing device 100, for example, via the network NT1. The network NT1 is typically a wireless line, but may include at least a wired line. The transmitter 183 may transmit the first observation information in real time, or may collectively transmit multiple pieces of first observation information generated at different times.

[0046] The first observation information is, for example, information about a reflected wave of an electromagnetic wave of a first frequency transmitted to a target area. In detail, the first observation information associates, for example, one or more of the intensity, observation position, observation direction, observation time, etc. of the reflected wave.

[0047] The observation position is a position in real space where observation is performed, and may be at least one of, for example, the transmission position of the electromagnetic wave of the first frequency, the reception position of the reflected wave, a position obtained using the transmission position and the reception position, such as a position midway between the transmission position and the reception position, etc. The observation position is expressed, for example, by latitude, longitude, height, etc., and may be obtained by providing the mobile body 180 with a GPS (Global Positioning System) function. Note that the observation position is not limited to the example given here.

[0048] The observation direction may be at least one of the transmission direction of the electromagnetic wave of the first frequency, the reception direction of the reflected wave, a direction obtained using the transmission direction and the reception direction, etc. Furthermore, the receiving unit 182 may include one or more antennas to obtain the reception direction.

[0049] The observation time is information indicating the time of observation, such as the observation time. The observation time may be at least one of the time when the electromagnetic wave of the first frequency is transmitted (e.g., the transmission time), the time when the reflected wave is received (e.g., the reception time), a time associated with the transmission time and the reception time, such as midway between the transmission time and the reception time, etc. The observation time may be acquired by providing the mobile object 180 with a timekeeping function.

[0050] (Regarding the Imaging Device 190) The imaging device 190 is a device for capturing an image of a target area using light, which is an electromagnetic wave of a second frequency, to generate second observation information.

[0051] As described above, the light includes visible light and infrared light. For example, when visible light is used, the image capturing device 190 is a visible light camera. For example, when infrared light, near-infrared light, or far-infrared light is used, the image capturing device 190 is an infrared camera, a near-infrared camera, or a far-infrared camera, respectively.

[0052] The second observation information is information indicating the results of observing the target area using electromagnetic waves of the second frequency. If the target area includes a predetermined area on the earth's surface, the second observation information is information indicating the results of observing the earth's surface using electromagnetic waves of the second frequency. If the electromagnetic waves of the second frequency are light, the observation using the electromagnetic waves of the second frequency corresponds to photography, and the second observation information includes a photographed image photographed by the photographing device 190.

[0053] For example, the image capturing device 190 generates second observation information including a captured image of the target area, and transmits the captured image to the information processing device 100 via the network NT2, for example.

[0054] The captured image is, for example, a color image such as an RGB image, but may also be a monochrome image.

[0055] The network NT2 is typically a wireless network, but may include at least a wired network. The networks NT1 and NT2 may be partly or entirely a common network, or partly or entirely different networks.

[0056] The image capturing device 190 may be fixed in position, or may be mounted on the above-described mobile body 110 or a mobile body different from the mobile body 110. This mobile body may be an aerial vehicle such as a drone, or a vehicle such as an automobile. In addition, this mobile body may be remotely controlled or operated by an operator, or may move automatically according to a predetermined algorithm or the like.

[0057] When the image capturing device 190 is mounted on a moving object, the image capturing device 190 may capture images of the target area while moving. In this case, the image capturing device 190 may transmit captured images in real time, or may transmit multiple images captured at different times together.

[0058] The image capturing device 190 may transmit second observation information in which at least one of the image capturing location and the image capturing time is associated with the captured image.

[0059] The shooting position is the position in real space where the image was captured. The shooting position is expressed, for example, by latitude, longitude, and altitude, and may be acquired by providing the image capturing device 190 or a mobile object equipped with the same with a GPS (Global Positioning System) function. Note that the shooting position is not limited to the example given here.

[0060] The photographing period is the period when the photograph was taken, for example, the time of the photograph. Note that the photographing period is not limited to the example given here.

[0061] The first observation information and the second observation information may be obtained at different times. However, it is desirable that the observation conditions (e.g., target area, condition of the target area, weather, etc.) excluding the observation time are generally similar. For example, for an area where the presence of an object is indicated in either the first or second observation information, the object may be identified by acquiring information from the other information using either the first or second observation information. This allows for more efficient identification of objects in a wider target area than by constantly processing both the first and second observation information.

[0062] (Detailed example of information processing device 100) As shown in FIG. 4 , the information processing device 100 may include an output control unit 150 and a display unit 160 in addition to a first acquisition unit 110, a second acquisition unit 120, an integration unit 130, and an identification unit 140.

[0063] The output control unit 150 outputs a target area map showing the existence area in the target area.

[0064] The display unit 160 is an example of an output destination of the output control unit 150. In this case, the output control unit 150 may cause the display unit 160 to display the target area map.

[0065] The display unit 160 may be provided in a device other than the information processing device 100, which is connected to the information processing device 100 via a network configured, for example, by wire, wirelessly, or a combination of these, so that information can be transmitted and received between the display unit 160 and the information processing device 100. The output destination of the output control unit 150 is not limited to the display unit, and may be, for example, a storage unit provided in the information processing device 100 or another device.

[0066] The information processing device 100 may execute information processing as shown in FIG.

[0067] Steps S110 to S140 are as described above.

[0068] The output control unit 150 outputs a target area map showing the existence area in the target area (step S150).

[0069] (First Acquisition Unit 110) A detailed example of the first acquisition unit 110 and the first acquisition process (step S110) executed by the first acquisition unit 110 will be described.

[0070] The first acquisition unit 110 acquires the first information based on the first observation information generated using electromagnetic waves of the first frequency, for example. As described above, the first information is information that associates the position of an object in a target region with a first reliability that indicates the likelihood that the object is present at that position.

[0071] In detail, for example, the first acquisition unit 110 generates three-dimensional information about the three-dimensional point cloud based on the first observation information transmitted from the transmission unit 183, and generates first information about the two-dimensional point cloud by converting the three-dimensional point cloud. The first acquisition unit 110 includes, for example, a first observation information acquisition unit 111, a three-dimensional information acquisition unit 112, and a projection unit 113, as shown in FIG.

[0072] The first observation information acquisition unit 111 acquires the first observation information from the transmission unit 183 .

[0073] The three-dimensional information acquisition unit 112 acquires three-dimensional information relating to the three-dimensional point cloud based on the first observation information acquired by the first observation information acquisition unit 111 .

[0074] The projection unit 113 uses the three-dimensional information acquired by the three-dimensional information acquisition unit 112 to generate first information about a two-dimensional point group obtained by projecting the three-dimensional point group onto a projection plane.

[0075] The first acquisition unit 110 executes a first acquisition process (step S110) as shown in FIG.

[0076] The first observation information acquisition unit 111 acquires the first observation information from the transmission unit 183 (step S111).

[0077] The three-dimensional information acquisition unit 112 acquires three-dimensional information related to the three-dimensional point cloud based on the first observation information acquired by the first observation information acquisition unit 111 (step S112).

[0078] The three-dimensional point cloud is a point cloud in three-dimensional space corresponding to a target area. For example, as described above, the target area is an area including the earth's surface, and may include above ground and underground. Therefore, the three-dimensional point cloud may include a point cloud representing at least one of the above ground areas, the underground area, and the above ground area. Note that, as described above, the target area may be within an object, such as the inside of a wall of a building, and in this case, the three-dimensional point cloud may include a point cloud representing the inside of the object, such as the inside of a wall of a building.

[0079] The three-dimensional information acquisition unit 112 may, for example, use the first observation information to calculate the distance and angle of the reflection point by fast Fourier transform (FFT) and acquire a three-dimensional point cloud within the target region. Furthermore, for example, the three-dimensional information acquisition unit 112 may calculate a first reliability for each three-dimensional point cloud based on the first observation information. This allows the three-dimensional information acquisition unit 112 to generate three-dimensional information in which the three-dimensional point cloud and the first reliability are associated with each other.

[0080] The first reliability is a value corresponding to the likelihood that an object exists in the associated three-dimensional point cloud. The first reliability may be calculated based on the intensity of the reflected wave in the three-dimensional point cloud, and may be a value indicating, for example, the probability that an object exists in the three-dimensional point cloud. The first reliability may have a larger value, for example, as the object exists at each point in the three-dimensional space.

[0081] In more detail, for example, the three-dimensional point cloud may be represented by identification information for identifying each of the three-dimensional point clouds and the position of each of the three-dimensional point clouds. In this case, the three-dimensional information is information in which the identification information for identifying each of the three-dimensional point clouds, the position of each of the three-dimensional point clouds, and the first reliability are associated with each other.

[0082] If the target region is three-dimensional, for example, three-dimensional information may be used as the first information. Furthermore, there may be a plurality of pieces of first observation information. In this case, the plurality of pieces of first observation information may be generated and transmitted by each of the plurality of mobile objects 180. Furthermore, the mobile object 180 may include a plurality of pairs of transmitters 181 and receivers 182. The plurality of pieces of first observation information may be generated by each of the plurality of pairs of transmitters 181 and receivers 182 and transmitted from one or a plurality of transmitters 183.

[0083] The projection unit 113 generates first information related to the two-dimensional point cloud, for example, by transforming the three-dimensional point cloud (step S113).

[0084] This transformation is a projection onto a predetermined projection plane. That is, the two-dimensional point cloud is a point cloud obtained by projecting a three-dimensional point cloud onto a projection plane. FIG. 8 is a diagram showing an example of projecting a three-dimensional point cloud onto a two-dimensional point cloud on a projection plane. FIG. 8(a) is a diagram showing an example of a projection plane onto which the three-dimensional point cloud is projected. FIG. 8(b) is a diagram showing an example of first information generated by the projection. In this case, the first reliability may be associated with the position of each point cloud.

[0085] The projection plane is an example of a two-dimensional target region, and is, for example, a plane corresponding to the earth's surface or a plane parallel to the earth's surface at a predetermined distance above or below the earth's surface.

[0086] Such a conversion from a three-dimensional point cloud to a two-dimensional point cloud may be performed using a general technique for transforming an image, such as affine transformation, homography transformation, etc. Note that the conversion from a three-dimensional point cloud to a two-dimensional point cloud is not limited to a general technique for transforming an image, and may also use, for example, a technique for transforming a coordinate system.

[0087] The projection plane is not limited to the one exemplified here, and may be defined, for example, not parallel to the earth's surface, and may be defined using altitude or the like instead of distance to the earth's surface.

[0088] For example, the first information is information for associating, for each of the two-dimensional point groups, at least a portion of the following: identification information for identifying each of the two-dimensional point groups, a position on the projection plane, a corresponding three-dimensional point group, and a first reliability, which is a value corresponding to the likelihood that an object exists.

[0089] The identification information for identifying each of the two-dimensional point clouds may be automatically assigned according to a predetermined rule, for example. However, the method for assigning the identification information is not limited to this, and some or all of the information may be manually input.

[0090] The corresponding three-dimensional point group is information about the three-dimensional point group of the projection source.

[0091] For example, the corresponding 3D point clouds may be identification information for identifying each of the 3D point clouds, and by using the identification information and the 3D information, each of the 2D point clouds can be associated with each piece of information included in the 3D information.

[0092] Furthermore, for example, the corresponding three-dimensional point cloud may include one or more of the information included in the three-dimensional information, i.e., identification information for identifying each of the three-dimensional point clouds, the position of each of the three-dimensional point clouds, and the reliability thereof. In this way, the first information directly includes the information of the corresponding three-dimensional point cloud, thereby making it possible to associate information about the three-dimensional point cloud from which the projection is made.

[0093] The first reliability included in the first information may be set based on the first reliability assigned to the corresponding three-dimensional point cloud. For example, the first reliability included in the first information may be the same as the first reliability assigned to the corresponding three-dimensional point cloud.

[0094] However, when projecting onto a projection plane, a plurality of three-dimensional point groups may be projected onto a common position on the projection plane.

[0095] In such cases, the corresponding 3D point cloud may be the plurality of 3D point clouds, or may be a 3D point cloud that is representative of the plurality of 3D point clouds (e.g., the 3D point cloud closest to the projection plane, the 3D point cloud associated with the highest confidence, etc.).

[0096] Furthermore, when the corresponding three-dimensional point cloud is a plurality of three-dimensional point clouds, the first reliability included in the first information may be, for example, an average value of the first reliabilities associated with the plurality of three-dimensional point clouds. When the corresponding three-dimensional point cloud is a three-dimensional point cloud closest to the projection plane, the first reliability included in the first information may be, for example, the first reliability associated with the closest three-dimensional point cloud. When the corresponding three-dimensional point cloud is a three-dimensional point cloud associated with the largest first reliability, the first reliability included in the first information may be, for example, the first reliability of the three-dimensional point cloud.

[0097] In the first information generated as a result of the conversion, the projection plane may be partitioned (divided) into a plurality of small regions using a mesh structure, as shown in FIG. 9 . While the drawing shows an example in which the small regions are square, the shape, number, etc. of the small regions may be changed as appropriate. In this case, the first reliability may be associated with each small region. The first reliability associated with each small region may be determined by statistically processing the first reliability included in the small region. In detail, for example, the first reliability associated with each small region may be an average value, a maximum value, a minimum value, etc.

[0098] (Second Acquisition Unit 120) A detailed example of the second acquisition unit 120 and the second acquisition process (step S120) executed by the second acquisition unit 120 will be described.

[0099] The second acquisition unit 120 acquires the second information based on, for example, second observation information generated using electromagnetic waves of the second frequency. As described above, the second information is information that associates the position of an object in a target region with a second reliability that indicates the likelihood that the object is present at that position.

[0100] In detail, for example, the second acquisition unit 120 generates the second information by detecting an area where the object exists in the two-dimensional captured image based on the captured image of the target area transmitted from the imaging device 190. Note that if the second observation information is not a captured image, the second acquisition unit 120 and the second acquisition process (step S120) may include the same functions and steps as the first acquisition unit 110 and the first acquisition process (step S110) described above, respectively.

[0101] The two-dimensional captured image is an image obtained by deforming the captured image of the target area so that the coordinate system representing the captured image of the target area is the same as the coordinate system representing the two-dimensional point cloud in the first information. In other words, the two-dimensional captured image can also be said to be a captured image of the scanning area represented in a coordinate system common to the two-dimensional point cloud. Note that although the captured image may also be a two-dimensional image, in this disclosure, in order to distinguish between images before and after deformation, the image before deformation is referred to as the captured image and the image after deformation is referred to as the two-dimensional captured image.

[0102] That is, the second acquisition unit 120 transforms, for example, the captured image transmitted from the imaging device 190 into a two-dimensional captured image as needed, and generates second information that associates the position of the object in the two-dimensional captured image with the second reliability. As shown in FIG. 10 , the second acquisition unit 120 includes, for example, a second observation information acquisition unit 121, an image transformation unit 122, and a second information generation unit 123.

[0103] The second observation information acquisition unit 121 acquires second observation information from the image capturing device 190 .

[0104] The image transformation unit 122 transforms the captured image included in the second observation information into a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud.

[0105] The second information generator 123 generates second information using the two-dimensional captured image.

[0106] The second acquisition unit 120 executes a second acquisition process (step S120) as shown in FIG.

[0107] The second observation information acquisition unit 121 acquires second observation information from the image capturing device 190 (step S121).

[0108] The image transformation unit 122 transforms the captured image included in the second observation information into a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud (step S122).

[0109] For example, the image deformation unit 122 deforms the captured image into a two-dimensional captured image so that the positions in real space of pixels included in the captured image coincide with the positions in real space of the two-dimensional point cloud.

[0110] The process of step S122 may be performed as needed. For example, the process of step S122b may be performed when the coordinate systems of the captured image acquired in step S121 and the two-dimensional point cloud included in the first information generated in step S113 are different, but may not be performed when they are the same.

[0111] The second information generator 123 generates the second information using the two-dimensional captured image (step S123).

[0112] In detail, for example, the second information generator 123 detects an object included in the two-dimensional captured image, calculates a second reliability for the detected object, and generates second information including the position and second reliability of the detected object. Various general techniques may be used as a technique for generating such second information.

[0113] Here, an example will be described in which an image analysis model is used, which is a machine learning model that has been trained to generate second information from an image using visible light or infrared light. Note that various models such as a Support Vector Machine (SVM), a Deep Neural Network (DNN), or a Convolutional Neural Network (CNN) may be used as the machine learning model, and the same applies hereinafter.

[0114] The image using visible light or infrared light is, for example, a two-dimensional captured image, but may also be a captured image in cases where a two-dimensional captured image is not generated as described above, etc. The image analysis model is, for example, a model that, when an image using visible light or infrared light is input, detects an object from the image, calculates a second reliability for the detected object, and outputs second information including the position and second reliability of the detected object.

[0115] The second information generator 123 may input an image using electromagnetic waves of the second frequency to the image analysis model to generate the second information. In training the image analysis model, training data including images using visible light or infrared light and positions and labels of objects may be used.

[0116] 12 and 13 are diagrams showing an example and another example of the second information, respectively. FIG. 12 illustrates objects Qa to Qc detected in a two-dimensional captured image and their respective positions. In this case, the second reliability may be associated with, for example, each pixel value. FIG. 13 shows an example in which a two-dimensional captured image is partitioned (divided) into multiple small regions using a mesh structure similar to that of FIG. 9. In this case, the second reliability may be associated with each small region.

[0117] (Regarding the Integration Unit 130) As described above, the integration unit 130 generates reliability map information based on the first reliability and the second reliability for each of the sub-regions corresponding to each position in the target region. For example, the integration unit 130 generates the reliability map information by integrating the first reliability and the second reliability for each of the sub-regions corresponding to each position in the target region.

[0118] As described above, the reliability map information is information that indicates the reliability of each sub-region. For example, the reliability map information is information that associates each sub-region with the reliability after integration.

[0119] The integration unit 130, for example, uses the first reliability and second reliability of the two-dimensional point cloud and pixel located at a position corresponding to the sub-region set in the target region to calculate the reliability after integration for each sub-region, as shown in Figure 14.

[0120] 14 is a diagram illustrating an example of reliability map information generated by integrating the first information (two-dimensional point cloud) and the second information (two-dimensional captured image). This figure illustrates an example in which each position of the two-dimensional point cloud included in the target region is treated as a subregion. In this case, it is preferable to associate each position of the two-dimensional point cloud with a reliability after integration.

[0121] There are various methods for calculating the reliability after integration (i.e., the reliability for each sub-region). Examples of the reliability after integration include a simple average, a weighted average, a median, a maximum value, a minimum value, and a top-k average. That is, the integration unit 130 may calculate at least one of the simple average, the weighted average, the median, the maximum value, the minimum value, and the top-k average of the first reliability and the second reliability for each sub-region.

[0122] Here, the top-k average is the average of k values ​​taken from M values ​​(M is an integer of 2 or greater) in descending order of value. k is a value that can be arbitrarily set between 1 and M. When k=1, the top-k average is the maximum value of the M values. When k=M, the top-k average is the simple average of all M values.

[0123] For example, suppose that a certain sub-region has a total of M first reliabilities and second reliabilities. In this case, the top-k average of the sub-region is, for example, the average value of the k first reliabilities and second reliabilities extracted by sorting the M first reliabilities and second reliabilities in descending order.

[0124] The sub-regions are not limited to two-dimensional point clouds, but may be appropriately defined regions such as pixels or small regions as illustrated in Figures 9 and 13. In this case, the reliability map information may also associate the reliability after integration with each sub-region.

[0125] The integrating unit 120 may input the first reliability and the second reliability for each sub-region into a function such as a linear function or a nonlinear function to calculate the reliability after integration for each sub-region. Such a function may be set in advance by, for example, referring to a reliability map.

[0126] (Target Area Map) As described above, the identification unit 140 processes the reliability map information to identify the area in which the target object exists. Then, the output control unit 150 may, for example, cause the display unit 160 to display the target area map.

[0127] 15 is a diagram showing an example of a target area map. The target area map shown in the figure includes reliability map information. The target area map shown in the figure also shows an example in which a contour image is used to indicate the area in which the identified object exists. Note that the target area map is not limited to this, and for example, a frame of a predetermined shape (e.g., a rectangle) surrounding the area in which the identified object exists may be used to indicate the area in which the identified object exists.

[0128] (Example of Physical Configuration of Information Processing Device 100) The information processing device 100 is, for example, a general-purpose computer. The information processing device 100 includes, for example, a bus 1010, a processor 1020, a memory 1030, a storage device 1040, a network interface 1050, an input interface 1060, and an output interface 1070, as shown in FIG. 16 .

[0129] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, network interface 1050, input interface 1060, and output interface 1070. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0130] The processor 1020 is implemented as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0131] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0132] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules for realizing the functions of the information processing apparatus 100 that includes the storage device 1040. The processor 1020 reads each of these program modules into the memory 1030 and executes them to realize the function corresponding to the program module.

[0133] The network interface 1050 is an interface for connecting the information processing device 100 equipped with the network interface 1050 to the networks NT1 and NT2.

[0134] The input interface 1060 is an interface for the user to input information, and is configured from, for example, a touch panel, a keyboard, a mouse, and the like.

[0135] The output interface 1070 is an interface for presenting information to the user, and is configured, for example, by a liquid crystal panel, an organic EL (Electro-Luminescence) panel, or the like.

[0136] Note that the physical configuration of the information processing device 100 is not limited to this. For example, the information processing device 100 may be composed of multiple devices. In this case, each device may be, for example, a computer or the like having a physical configuration similar to that of the information processing device 100 shown in FIG. 16 .

[0137] As described above, according to this embodiment, the reliability for each sub-region includes at least one of the simple average, weighted average, median, maximum value, minimum value, and top-k average of the first reliability and second reliability for each sub-region.

[0138] By using such integrated reliability, it is possible to identify an object by comprehensively considering the reliability obtained using electromagnetic waves of different frequencies, thereby enabling accurate identification of the object from information generated using electromagnetic waves.

[0139] According to the present embodiment, the electromagnetic waves of the second frequency are visible light or infrared light. The second acquisition unit 120 generates the second information by inputting the image using the electromagnetic waves of the second frequency to an image analysis model that has been trained to generate the second information from an image using visible light or infrared light.

[0140] This allows the target to be identified by comprehensively considering the second reliability obtained from an image using visible light or infrared light and the first reliability obtained using electromagnetic waves of a different frequency from visible light or infrared light, thereby making it possible to accurately identify the target from information generated using electromagnetic waves.

[0141] According to this embodiment, the target area includes a predetermined area on the earth's surface. The first information is information generated using first observation information indicating the results of observing the earth's surface using a radar mounted on a moving object and irradiating electromagnetic waves of a first frequency. The second information is information generated using second observation information indicating an image of the earth's surface captured using a camera that captures images using visible light or infrared light.

[0142] This makes it possible to identify objects that exist near the ground surface by comprehensively considering the reliability obtained using electromagnetic waves of different frequencies. Therefore, it becomes possible to identify objects observed from the sky with high accuracy. Therefore, it becomes possible to identify objects that exist near the ground surface with high accuracy.

[0143] According to this embodiment, the moving body is an air vehicle.

[0144] This allows the target object to be identified using the results of observation from the sky. Therefore, the target object can be identified with high accuracy using the results of observation from the sky.

[0145] Second Embodiment The integration unit 130 may generate the reliability map information using a machine learning model.

[0146] In this case, the integration unit 130 generates reliability map information by inputting the first reliability and the second reliability for each sub-region into an integrated model that has been trained to generate reliability map information for each sub-region, for example.

[0147] The integrated model may be trained using training data including the first and second reliabilities for each subregion and corresponding correct reliability map information. The integration unit 130 may then use the integrated model trained using such training data. The information processing device 100 may further include an integrated model training unit (not shown) that trains the integrated model.

[0148] As described above, according to this embodiment, the integration unit 130 inputs the first reliability and the second reliability for each sub-region into an integrated model that has been trained to generate reliability map information for each sub-region, and generates reliability map information.

[0149] In this way, by using the integrated reliability to identify the area where the object exists, the object can be identified by comprehensively considering the reliability obtained using electromagnetic waves of different frequencies, thereby making it possible to accurately identify the object from information generated using electromagnetic waves.

[0150] Third Embodiment The identification unit 140 may identify an area where an object exists in a target area by processing the reliability map information using a threshold. This threshold may be set manually or automatically using statistical processing or the like. An example of automatically setting a threshold and identifying an area where an object exists using this threshold will be described below.

[0151] 17, the specifying unit 140a includes a threshold setting unit 141 and an existence region specifying unit 142. The specifying unit 140a is a detailed example of the specifying unit 140.

[0152] The threshold setting unit 141 performs statistical processing on the reliability map information to set a threshold according to the area where the object exists.

[0153] The existence region identifying unit 142 identifies the existence region of the object by processing the reliability map information using a threshold value.

[0154] The identification unit 140a executes an identification process (step S140a) as shown in Fig. 18. This identification process (step S140a) is a detailed example of the above-mentioned identification process (step S140).

[0155] The threshold setting unit 141 performs statistical processing on the reliability map information to set a threshold according to the area where the object exists (step S141).

[0156] For example, the threshold setting unit 141 uses statistics such as the average value and variance of the reliability after integration to set a value that falls within a confidence interval with a significance level of 5% as the threshold.

[0157] The method of setting a threshold value by performing statistical processing on the reliability map information is not limited to this example.

[0158] The existence region specifying unit 142 specifies the existence region of the object by processing the reliability map information using a threshold value (step S142).

[0159] For example, the existence region identifying unit 142 may compare the reliability after integration with a threshold and identify the existence region of the object based on the comparison result. In detail, for example, the identifying unit 140 may identify the existence region of the object as a region where the reliability after integration is greater than the threshold or a region where the reliability after integration is equal to or greater than the threshold.

[0160] As described above, according to this embodiment, the identification unit 140a includes a threshold setting unit 141 and an existence region identification unit 142. The threshold setting unit 141 performs statistical processing on the reliability map information to set a threshold according to the existence region of the object. The existence region identification unit 142 processes the reliability map information using the threshold to identify the existence region of the object.

[0161] This allows the threshold to be automatically set and the area where the object exists to be identified using the threshold, thereby eliminating the need for a person to manually set the threshold. As a result, it becomes possible to accurately and easily identify the object from information generated using electromagnetic waves.

[0162] Furthermore, by using statistical processing, it is possible to set an appropriate threshold and more accurately identify the area where the target exists, which makes it possible to more accurately identify the target from information generated using electromagnetic waves.

[0163] [Embodiment 4] The identification unit 140 may identify the presence area of ​​the object in the target area by processing the reliability map information using previously prepared correct answer data instead of the threshold described in embodiment 3. The correct answer data is, for example, data created based on observation information obtained by previously observing an object whose shape, etc. corresponds to that of the object using the transmitting unit 181 and the receiving unit 182. The correct answer data includes, for example, a two-dimensional point cloud based on the observation information of the object.

[0164] The identification unit 140 performs matching using, for example, the supervised data and the reliability map information. In this matching, for example, the supervised data and the reliability map information are aligned so that the point groups included in each of them are most compatible according to a predetermined scale. At this time, it is preferable to perform alignment by converting the coordinate axis system of the supervised data, for example.

[0165] The identification unit 140 may then identify a match region including a point group included in the correct answer data after matching as a region where the object exists. The match region is, for example, a region within the outline image described above, but is not limited to this. Furthermore, among the point groups indicated by the reliability map information, a point group within the match region may be identified as a region where the object exists.

[0166] The correct answer data is not limited to data created based on observation information. The correct answer data may include, for example, an approximation model showing the external shape of an object whose shape, etc., corresponds to that of the target object. The approximation model shows, for example, a shape represented by an image showing the above-mentioned match area. In this case, when matching the correct answer data with the reliability map information, for example, an approximation point cloud may be generated by sampling from the approximation model, and this pseudo point cloud may be used instead of the two-dimensional point cloud based on the above-mentioned observation information.

[0167] As described above, according to this embodiment, the identification unit 140 identifies the area in which the target object exists by processing the reliability map information using correct answer data created based on observation information obtained by previously observing an object whose shape, etc. corresponds to that of the target object.

[0168] In this embodiment, similarly to the first embodiment, the reliability after integration is used to identify the area where the object exists, so that the object can be identified by comprehensively considering the reliability obtained using electromagnetic waves of different frequencies. Therefore, it becomes possible to identify the object with high accuracy from information generated using electromagnetic waves.

[0169] Fifth Embodiment The identification unit 140 may use a machine learning model to identify an area in the target area where an object exists.

[0170] The information processing device 100 may include, for example, a first acquisition unit 110, a second acquisition unit 120, an integration unit 130, an identification unit 140b, an output control unit 150, and a learning unit 170, as shown in FIG.

[0171] The identification unit 140b inputs the reliability map information generated by the integrating unit 130 into an object identification model, which is a machine learning model that has learned to identify an object existence area from the reliability map information, to identify an object existence area. The identification unit 140b is another detailed example of the identification unit 140, and executes processing equivalent to step S140.

[0172] The learning unit 170 learns the object identification model using training data prepared in advance. This training data is prepared in advance for learning and may include, for example, shape information in addition to confidence map information for learning (the position of the object and the confidence after integration). The shape information is information related to the shape of the object, and is, for example, information indicating at least one of the object's contour, appearance, outer shape, etc. The information processing performed by the information processing device 100 may further include such processing (learning processing) performed by the learning unit 170.

[0173] As described above, according to this embodiment, the identification unit 140b inputs the reliability map information to an object identification model that has been trained to identify the existence area of ​​the object from the reliability map information, and identifies the existence area of ​​the object.

[0174] This allows the use of a machine learning model to identify the area where an object exists without the need for a human to set a threshold, making it possible to accurately and easily identify an object from information generated using electromagnetic waves.

[0175] According to this embodiment, the information processing device 100 includes a learning unit 170 that uses prepared training data to train a machine learning model. The training data includes shape information relating to the shape of an object.

[0176] This allows learning using shape information, eliminating the need for a human to set a threshold and increasing the likelihood of accurately identifying the area where an object exists. This makes it possible to more accurately and easily identify an object from information generated using electromagnetic waves.

[0177] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0178] In addition, although the flowcharts used in the above description show a sequence of steps (processes), the order of steps executed in each embodiment is not limited to the sequence shown in the flowcharts. In each embodiment, the order of steps shown in the diagrams can be changed as long as it does not cause any problems in terms of the content.

[0179] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0180] 1. An information processing device comprising: a first acquisition means for acquiring first information generated using electromagnetic waves of a first frequency and associating a position in a target area with a first reliability indicating the likelihood that an object exists at that position; a second acquisition means for acquiring second information generated using electromagnetic waves of a second frequency different from the first frequency and associating a position in the target area with a second reliability indicating the likelihood that an object exists at that position; an integration means for generating reliability map information indicating the reliability of each of the sub-areas corresponding to each position in the target area based on the first reliability and the second reliability for the sub-areas; and an identification means for identifying an existence area of ​​the object by processing the reliability map information. 2. The information processing device described in 1., wherein the identification means includes: a threshold setting means for setting a threshold according to the existence area of ​​the object by performing statistical processing on the reliability map information; and an existence area identification means for identifying the existence area of ​​the object by processing the reliability map information using the threshold. 3. The information processing device described in 1., wherein the identification means inputs the reliability map information into an object identification model that has been trained to identify an existence region of the object from the reliability map information, thereby identifying the existence region of the object. 4. The information processing device described in 3., further comprising learning means that trains a machine learning model using training data prepared in advance, wherein the training data includes shape information related to the shape of the object. 5. The information processing device described in any one of 1. to 4., wherein the reliability for each sub-region includes calculating at least one of a simple average, a weighted average, a median, a maximum value, a minimum value, and a top-k average of the first reliability and the second reliability for each sub-region. 6. The information processing device described in any one of 1. to 4., wherein the integration means inputs the first reliability and the second reliability for each sub-region into an integrated model that has been trained to generate the reliability map information for each sub-region, thereby generating the reliability map information.7. The information processing device described in any one of 1. to 6., wherein the electromagnetic waves of the second frequency are visible light or infrared light, and the second acquisition means generates the second information by inputting an image using the electromagnetic waves of the second frequency into an image analysis model that has been trained to generate the second information from an image using visible light or infrared light. 8. The information processing device described in any one of 1. to 7., wherein the target area includes a predetermined area within a predetermined range from the earth's surface, and the first information is information generated using first observation information that indicates a result of observing the earth's surface using a radar that is mounted on a moving body and irradiates electromagnetic waves of the first frequency, and the second information is information generated using second observation information that indicates an image of the earth's surface captured by a camera that captures images using visible light or infrared light. 9. The information processing device described in 8., wherein the moving body is an air vehicle. 10. The information processing device described in any one of 1. to 9., further comprising output control means for outputting a target area map that indicates the existence area in the target area. 11. an information processing system comprising: a mobile body that moves to generate first observation information by scanning the target area using a radar that irradiates electromagnetic waves of the first frequency; an imaging device that images the target area using electromagnetic waves of the second frequency to generate second observation information; and the information processing device described in any one of 1. to 10., wherein the mobile body includes: a transmitting means that transmits the electromagnetic waves of the first frequency to the target area; a receiving means that receives reflected waves of the transmitted electromagnetic waves of the first frequency and generates the first observation information related to the reflected waves; and a transmitting means that transmits the generated first observation information.12. An information processing method in which one or more computers acquire first information generated using electromagnetic waves of a first frequency, associating a position in a target area with a first reliability indicating the likelihood that an object is present at that position, acquire second information generated using electromagnetic waves of a second frequency different from the first frequency, associating a position in the target area with a second reliability indicating the likelihood that an object is present at that position, generate reliability map information indicating the reliability of each of the sub-areas corresponding to each position in the target area based on the first reliability and the second reliability for the sub-areas, and identify the area in which the object exists by processing the reliability map information. 13. The information processing method described in 12., in which identifying the area in which the object exists includes performing statistical processing on the reliability map information to set a threshold corresponding to the area in which the object exists, and processing the reliability map information using the threshold to identify the area in which the object exists. 14. The information processing method according to 12., wherein identifying the existence region of the object includes inputting the reliability map information into an object identification model that has been trained to identify the existence region of the object from the reliability map information, thereby identifying the existence region of the object. 15. The information processing method according to 14., further including training a machine learning model using training data prepared in advance, wherein the training data includes shape information related to the shape of the object. 16. The information processing method according to any one of 12. to 15., wherein the reliability for each sub-region includes calculating at least one of a simple average, a weighted average, a median, a maximum value, a minimum value, and a top-k average of the first reliability and the second reliability for each sub-region. 17. The information processing method according to any one of 12. to 15., wherein generating the reliability map information includes inputting the first reliability and the second reliability for each sub-region into an integrated model that has been trained to generate the reliability map information for each sub-region, thereby generating the reliability map information.18. The information processing method described in any one of 12. to 17., wherein the electromagnetic waves of the second frequency are visible light or infrared light, and acquiring the second information involves generating the second information by inputting an image using electromagnetic waves of the second frequency into an image analysis model that has been trained to generate the second information from an image using visible light or infrared light. 19. The information processing method described in any one of 12. to 18., wherein the target area includes a predetermined area within a predetermined range from the Earth's surface, the first information is information generated using first observation information that indicates a result of observing the Earth's surface using a radar that is mounted on a moving body and irradiates electromagnetic waves of the first frequency, and the second information is information generated using second observation information that indicates an image of the Earth's surface captured using a camera that captures images using visible light or infrared light. 20. The information processing device described in 19., wherein the moving body is an air vehicle. 21. The information processing method described in 12. to 20., further comprising outputting a target area map that indicates the existence area in the target area. 22. A program for causing one or more computers to execute the following: acquire first information, generated using electromagnetic waves of a first frequency, associating a position in a target area with a first reliability indicating the likelihood that an object is present at that position; acquire second information, generated using electromagnetic waves of a second frequency different from the first frequency, associating a position in the target area with a second reliability indicating the likelihood that an object is present at that position; generate reliability map information indicating the reliability of each sub-area corresponding to each position in the target area based on the first reliability and the second reliability for the sub-area; and identify an area where the object exists by processing the reliability map information. 23. The program described in 22., wherein identifying the area where the object exists includes: performing statistical processing on the reliability map information to set a threshold corresponding to the area where the object exists; and processing the reliability map information using the threshold to identify the area where the object exists.24. The program according to 22., wherein identifying the existence region of the object includes inputting the reliability map information into an object identification model that has been trained to identify the existence region of the object from the reliability map information, thereby identifying the existence region of the object. 25. The program according to 24., further including training a machine learning model using training data prepared in advance, wherein the training data includes shape information related to the shape of the object. 26. The program according to any one of 22. to 25., wherein the reliability for each subregion includes calculating at least one of a simple average, a weighted average, a median, a maximum value, a minimum value, and a top-k average of the first reliability and the second reliability for each subregion. 27. The program according to any one of 22. to 25., wherein generating the reliability map information includes inputting the first reliability and the second reliability for each subregion into an integrated model that has been trained to generate the reliability map information for each subregion, thereby generating the reliability map information. 28. The program described in any one of 22. to 27., wherein the electromagnetic waves of the second frequency are visible light or infrared light, and acquiring the second information involves inputting an image using electromagnetic waves of the second frequency into an image analysis model that has been trained to generate the second information from an image using visible light or infrared light, thereby generating the second information. 29. The program described in any one of 22. to 28., wherein the target area includes a predetermined area within a predetermined range from the Earth's surface, the first information is information generated using first observation information indicating results of observing the Earth's surface using a radar mounted on a moving object that irradiates electromagnetic waves of the first frequency, and the second information is information generated using second observation information indicating images of the Earth's surface captured by a camera that captures images using visible light or infrared light. 30. The program described in 29., wherein the moving object is an airborne object. 31. The program described in any one of 22. to 30., further causing the program to output a target area map indicating the existence area within the target area. 32. A recording medium on which the program according to any one of 22. to 31. is recorded.

[0181] This application claims priority based on Japanese Patent Application No. 2023-137121, filed on August 25, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0182] REFERENCE SIGNS LIST 100 Information processing device 110 First acquisition unit 111 First observation information acquisition unit 112 Three-dimensional information acquisition unit 113 Projection unit 120 Second acquisition unit 121 Second observation information acquisition unit 122 Image deformation unit 123 Second information generation unit 130 Integration unit 140, 140a, 140b Identification unit 141 Threshold setting unit 142 Existence area identification unit 150 Output control unit 160 Display unit 170 Learning unit 180 Mobile object 181 Transmission unit 182 Reception unit 183 Transmission unit 190 Imaging device

Claims

1. A first acquisition means that generates first information using electromagnetic waves of a first frequency and associates the position in the target region with a first confidence level indicating the likelihood that an object exists at that position, A second acquisition means generates electromagnetic waves of a second frequency different from the first frequency and acquires second information that associates the position in the target region with a second confidence level indicating the likelihood that an object exists at that position, An integration means for generating confidence map information indicating the confidence level of each sub-region based on the first confidence level and the second confidence level in each sub-region corresponding to each position in the target region, The system includes a means for identifying the area where an object exists by processing the aforementioned confidence map information. Information processing device.

2. The aforementioned specifying means is, A threshold setting means that sets a threshold corresponding to the area where the object exists by performing statistical processing on the confidence map information, The system includes means for identifying the area where the object exists by processing the confidence map information using the threshold value. The information processing apparatus according to claim 1.

3. The identification means inputs the confidence map information into an object identification model that has been trained to identify the object's location from the confidence map information, thereby identifying the object's location. The information processing apparatus according to claim 1.

4. Furthermore, it includes a learning method for training machine learning models using pre-prepared training data. The training data includes shape information relating to the shape of the object. The information processing apparatus according to claim 3.

5. The confidence level for each sub-region includes at least one of the following: the simple average of the first confidence level and the second confidence level for each sub-region, the weighted average, the median, the maximum value, the minimum value, and the top-k mean. The information processing apparatus according to any one of claims 1 to 4.

6. The integration means inputs the first confidence level and the second confidence level for each sub-region into an integrated model trained to generate the confidence map information, thereby generating the confidence map information. The information processing apparatus according to any one of claims 1 to 4.

7. The electromagnetic wave of the second frequency is visible light or infrared light. The second acquisition means inputs an image using electromagnetic waves of the second frequency into an image analysis model that has been trained to generate the second information from an image using visible light or infrared light, and generates the second information. The information processing apparatus according to any one of claims 1 to 4.

8. The aforementioned target area includes a predetermined area within a range set out from the ground surface. The first information is information generated using first observation information, which shows the results of observing the Earth's surface using radar mounted on a mobile body and irradiating electromagnetic waves of a first frequency. The second information is information generated using second observational information, which shows an image of the Earth's surface taken using a camera that takes photographs using visible light or infrared light. The information processing apparatus according to any one of claims 1 to 4.

9. One or more computers, First information is obtained by generating electromagnetic waves of a first frequency and associating the position in the target region with a first confidence level indicating the likelihood that an object exists at that position. It is generated using electromagnetic waves of a second frequency different from the first frequency, and second information is obtained that associates the position in the target region with a second confidence level indicating the likelihood that an object exists at that position. Based on the first and second confidence levels in each of the sub-regions corresponding to each position in the target region, confidence map information indicating the confidence level of each of the sub-regions is generated. By processing the aforementioned confidence map information, the region where the object exists can be identified. Information processing methods.

10. On one or more computers, First information is obtained by generating electromagnetic waves of a first frequency and associating the position in the target region with a first confidence level indicating the likelihood that an object exists at that position. It is generated using electromagnetic waves of a second frequency different from the first frequency, and second information is obtained that associates the position in the target region with a second confidence level indicating the likelihood that an object exists at that position. Based on the first and second confidence levels in each of the sub-regions corresponding to each position in the target region, confidence map information indicating the confidence level of each of the sub-regions is generated. A program for processing the aforementioned confidence map information to identify the region where an object exists.