Information processing device, information processing method, and program

Abstract

An information processing device according to the present disclosure comprises: a search condition acquisition unit that acquires a search condition in which a target object is expressed in text; a visible image processing unit that processes a visible image obtained by imaging a target region on the basis of the search condition and a language model to generate first reliability information indicating a region in which the target object may potentially be present in the target region; a reflected wave processing unit that processes reflected wave information indicating a reflected wave of electromagnetic waves irradiated onto the target region, to generate second reliability information indicating a region in which the target object may possibly be present in the target region; and an estimation unit that estimates a region in which the target object is present in the target region on the basis of the first reliability information and the second reliability information.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] A technique related to the present disclosure is disclosed in Patent Document 1. Patent Document 1 discloses a technique for detecting an object that reflects a large amount of millimeter-wave radar, such as metal, based on the reflection intensity of the millimeter-wave radar.

[0003] International Publication No. 2017 / 057058

[0004] The technology disclosed in Patent Document 1 detects all objects that reflect a lot of millimeter-wave radar, such as metals, etc. With the technology disclosed in Patent Document 1, it is difficult to accurately detect only a part of objects that reflect a lot of millimeter-wave radar.

[0005] In view of the above-described problems, an example of an object of the present disclosure is to provide an information processing device, an information processing method, and a program that accurately detect a desired object.

[0006] According to the present disclosure, there is provided an information processing device having: a search condition acquisition means for acquiring search conditions that express a target object in text; a visible image processing means for generating first reliability information that indicates an area within the target area where the target object may be present, by processing a visible image of a target area based on the search conditions and a language model; a reflected wave processing means for generating second reliability information that indicates an area within the target area where the target object may be present, by processing reflected wave information that indicates reflected waves of electromagnetic waves irradiated to the target area; and an estimation means for estimating an area within the target area where the target object is present, based on the first reliability information and the second reliability information.

[0007] The present disclosure also provides an information processing method in which one or more computers acquire search conditions that express a target object in text, process a visible image of a target area based on the search conditions and a language model to generate first reliability information that indicates an area of ​​the target area where the target object may be present, process reflected wave information that indicates reflected waves of electromagnetic waves irradiated to the target area to generate second reliability information that indicates an area of ​​the target area where the target object may be present, and estimate the area of ​​the target area where the target object is present based on the first reliability information and the second reliability information.

[0008] Furthermore, according to the present disclosure, there is provided a program that causes a computer to function as: a search condition acquisition means that acquires search conditions that express a target object in text; a visible image processing means that processes a visible image of a target area based on the search conditions and a language model to generate first reliability information that indicates an area within the target area where the target object may be present; a reflected wave processing means that processes reflected wave information that indicates reflected waves of electromagnetic waves irradiated to the target area to generate second reliability information that indicates an area within the target area where the target object may be present; and an estimation means that estimates an area within the target area where the target object is present based on the first reliability information and the second reliability information.

[0009] According to one aspect of the present disclosure, an information processing device, an information processing method, and a program for detecting a desired object with high accuracy are realized.

[0010] FIG. 1 is a diagram illustrating an example of a functional block diagram of an information processing device according to the present disclosure. FIG. 2 is a flowchart illustrating an example of a processing flow of an information processing device according to the present disclosure. FIG. 3 is a diagram illustrating an example of a hardware configuration of an information processing device according to the present disclosure. FIG. 4 is a diagram illustrating an example of processing performed by an information processing device according to the present disclosure. FIG. 5 is a diagram illustrating another example of processing performed by an information processing device according to the present disclosure. FIG. 6 is a diagram illustrating another example of processing performed by an information processing device according to the present disclosure.

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In this disclosure, the drawings relate to one or more embodiments. In all drawings, similar components are designated by similar reference numerals, and descriptions thereof will be omitted as appropriate.

[0012] First Embodiment Fig. 1 is a functional block diagram showing an overview of an information processing device 10. Fig. 2 is a flowchart showing an example of the flow of processing executed by the information processing device 10.

[0013] 1, the information processing device 10 includes a search condition acquisition unit 11, a visible image processing unit 12, a reflected wave processing unit 13, and an estimation unit 14. These functional units execute the process of FIG.

[0014] In S10, the search condition acquisition unit 11 acquires search conditions that express the target object in text.

[0015] In S11, the visible image processing unit 12 generates first reliability information. The first reliability information indicates an area of ​​the target area where the target object may exist. The first reliability information is generated by processing a visible image of the target area based on the search criteria and a language model.

[0016] In S12, the reflected wave processing unit 13 generates second reliability information. Like the first reliability information, the second reliability information indicates an area of ​​the target area where a target object may exist. The second reliability information is generated by processing reflected wave information that indicates the reflected wave of the electromagnetic wave irradiated to the target area.

[0017] In S13, the estimation unit 14 estimates an area in the target area where the target object exists, based on the first reliability information and the second reliability information.

[0018] According to the information processing device 10, which estimates the area of ​​the target area in which the target object exists based on "both" the first reliability information and the second reliability information as described above, the target object can be detected with high accuracy.

[0019] Second Embodiment Overview An information processing apparatus 10 according to a second embodiment is a specific implementation of the configuration of the information processing apparatus 10 according to the first embodiment. This will be described in detail below.

[0020] "Hardware Configuration" First, an example of the hardware configuration of the information processing device 10 will be described. Each functional unit of the information processing device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various variations in the realization method and device. Software includes programs that are pre-stored in the device before shipping, and programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet.

[0021] FIG. 3 is a block diagram illustrating an example of the hardware configuration of an information processing device 10. As shown in FIG. 3, the information processing device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The information processing device 10 does not necessarily have to have the peripheral circuit 4A. Note that the information processing device 10 may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices may have the above hardware configuration.

[0022] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to mutually transmit and receive data. The processor 1A is, for example, a processing unit such as a CPU or a graphics processing unit (GPU). The memory 2A is, for example, a random access memory (RAM) or a read-only memory (ROM). The input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. The input / output interface 3A also includes an interface for connecting to a communication network such as the Internet. Examples of input devices include a keyboard, mouse, microphone, physical buttons, touch panel, etc. Examples of output devices include a display, speaker, printer, mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0023] "Functional Configuration" Next, the functional configuration of the information processing device 10 will be described in detail. Fig. 1 shows an example of a functional block diagram of the information processing device 10. As shown in the figure, the information processing device 10 has a search condition acquisition unit 11, a visible image processing unit 12, a reflected wave processing unit 13, and an estimation unit 14.

[0024] The search condition acquisition unit 11 acquires search conditions that express target objects in text.

[0025] A "target object" is an object that is the target of a search. The information processing device 10 searches for the target object based on first reliability information generated by processing a visible image based on search conditions and a language model, and second reliability information generated by processing reflected wave information indicating reflected waves of electromagnetic waves. Therefore, objects that reflect electromagnetic waves can be the target of a search by the information processing device 10. Objects that reflect electromagnetic waves are often made mainly of metal.

[0026] The "search criteria" is text that expresses a target object. The search criteria acquired by the search criteria acquisition unit 11 become search criteria for object detection and object recognition based on the above-mentioned language model. The text that constitutes the search criteria is a word, a sentence, a prompt, etc. The text that constitutes the search criteria may include, for example, the type of object, the external characteristics of the object (color, size, shape, etc.), etc. An example of a search criteria is "blue can."

[0027] The search criteria may further include text expressing an object to be removed from the search results. That is, the search criteria may include text expressing a target object and text expressing an object to be removed from the search results. Hereinafter, the target object may be referred to as a "positive example," and the object to be removed from the search results may be referred to as a "negative example." The negative examples are expressed in the same manner as the positive examples. An example of a search criteria for negative examples is "white can."

[0028] The search condition acquisition unit 11 acquires search conditions input by a user. The user can input search conditions to the information processing device 10 via any input device such as a keyboard, a touch panel, a microphone, a mouse, or a physical button.

[0029] The visible image processing unit 12 processes the visible image captured of the target area based on the above search conditions and language model, thereby generating first reliability information indicating areas of the target area where the target object may exist.

[0030] The "target area" is an area to be searched for whether a target object exists. The target area may be a part of the ground, a part of a building, or other areas.

[0031] A "visible image" is an image generated by a visible camera. A visible camera is a camera that detects visible light and converts it into an image. A visible image may be a still image or a moving image. Information indicating the date and time of shooting and the location where the visible image was taken may be added to the visible image. If the visible image is a moving image, information indicating the date and time of shooting and the location where the image was taken may be added to each frame image.

[0032] In one example, the visible light camera is mounted on a measuring device. The measuring device is a mobile device. The measuring device may be, for example, an air vehicle such as a drone, or a self-propelled device on land. The measuring device may be mobile automatically or remotely controlled. For example, the measuring device may move automatically along a pre-registered route. As the measuring device moves, it captures images of the target area with the visible light camera.

[0033] In another example, the visible camera is carried by a worker who carries the visible camera and takes pictures of the target area while moving around.

[0034] The visible image generated by the visible camera is input to the information processing device 10 by any means. For example, the visible camera and the information processing device 10 may be configured to be able to communicate with each other. The visible camera may then transmit the generated visible image to the information processing device 10 via the communication means. The transmission of the visible image from the visible camera to the information processing device 10 may be performed by real-time processing or batch processing.

[0035] Alternatively, the visible images generated by the visible camera may be stored in any storage device. The storage device may be provided within the visible camera or in an external device configured to be able to communicate with the visible camera. The visible images stored in the storage device may then be input to the information processing device 10 at any timing and by any means.

[0036] "Processing based on search criteria and a language model" refers to object recognition that recognizes target objects indicated by search criteria based on an object recognition model using a language model, or object detection that detects target objects indicated by search criteria based on an object detection model using a language model. The object recognition model and object detection model are models that can represent images and language in a joint embedding space.

[0037] The object recognition model and object detection model are generated by learning the relationship between the results of object recognition / object detection obtained by a technology such as a neural network and language related to the objects (descriptions and expressions of the objects). The visible image processing unit 12 can recognize / detect objects indicated by the search criteria in visible images based on the object recognition model and object detection model. This technology is disclosed in, for example, the following documents: "Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021." "Li, Liunian Harold, et al. "Grounded language-image pre-training." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022."

[0038] The "first reliability information" indicates an area of ​​the target area in which the target object may exist.

[0039] An example of the first reliability information is shown in Figure 4. The first reliability information shown in the figure is obtained by dividing a visible image of a target area into multiple sub-areas, and indicates the reliability of the presence of a target object for each sub-area. The higher the reliability of a sub-area, the more likely it is that a target object exists. One sub-area may correspond to one pixel. Alternatively, one sub-area may correspond to multiple pixels.

[0040] By processing a visible image of a target region using an object recognition model or an object detection model that uses a language model, positive examples and negative examples indicated by the search criteria can be detected from the visible image. The visible image processing unit 12 can generate first reliability information based on the results of the processing and a reliability calculation rule prepared in advance. The details of the reliability calculation rule are not particularly limited. However, the reliability calculation rule is defined so as to relatively increase the reliability of a child region in which a positive example is detected and relatively decrease the reliability of a child region in which no positive example is detected or a child region in which a negative example is detected.

[0041] Returning to Figure 1, the reflected wave processing unit 13 processes reflected wave information indicating the reflected waves of the electromagnetic waves irradiated to the target area, thereby generating second reliability information indicating the area within the target area where the target object may be present.

[0042] The "electromagnetic waves" are, for example, millimeter waves, and an example of their wavelength is 0.3 GHz or more and 300 GHz or less. However, the band of the electromagnetic waves is not limited to millimeter waves. The electromagnetic waves may be near infrared rays, far infrared rays, etc.

[0043] "Reflected waves" are irradiated electromagnetic waves that are reflected by, for example, an object present in the target area. When an object that reflects electromagnetic waves is present in the target area, the strength of the reflected waves increases. Objects that reflect electromagnetic waves are often made of metal.

[0044] Such emission of electromagnetic waves and reception of reflected waves are achieved by an electromagnetic wave transmitting / receiving device. The electromagnetic wave transmitting / receiving device includes an electromagnetic wave transmitting unit that transmits electromagnetic waves and an electromagnetic wave receiving unit that receives reflected waves. The electromagnetic wave transmitting / receiving device may include multiple, for example, two, electromagnetic wave receiving units. These multiple electromagnetic wave receiving units are spaced apart from each other and receive reflected waves of the electromagnetic waves emitted by the same electromagnetic wave transmitting unit. This increases the accuracy of detecting the position of the target object.

[0045] The transmission method used by the electromagnetic wave transmitter is, for example, any of FMCW (Frequency Modulated Continuous Wave), pulse, CW (Continuous Wave) Doppler, two-frequency CW, and pulse compression, but may be other than these.

[0046] In one example, the measurement device equipped with the above-mentioned visible camera further includes an electromagnetic wave transmitting and receiving device. While the measurement device moves, the visible camera captures an image of a target area, and the electromagnetic wave transmitting and receiving device irradiates the target area with electromagnetic waves and receives the reflected waves.

[0047] In another example, the electromagnetic wave transmitting and receiving device is carried by a worker. The worker moves around carrying the visible light camera and the electromagnetic wave transmitting and receiving device, photographing the target area with the visible light camera, and irradiating the target area with electromagnetic waves using the electromagnetic wave transmitting and receiving device and receiving the reflected waves.

[0048] "Reflected wave information" is generated by the electromagnetic wave transmitting and receiving device. More specifically, the reflected wave information is generated based on the results of reception of the reflected wave by the electromagnetic wave receiving unit. The reflected wave information includes, for example, time series information on the intensity of the reflected wave. This time series information includes a combination of the date and time the reflected wave was received and the intensity of the reflected wave at that time. If multiple electromagnetic wave receiving units are provided, reflected wave information is generated for each of the multiple electromagnetic wave receiving units.

[0049] The electromagnetic wave transmitting and receiving device may generate location information indicating the location of the electromagnetic wave transmitting and receiving device. The location information is indicated by, for example, latitude and longitude. The location information may include altitude in addition to latitude and longitude. This location information may be generated using, for example, GPS, or may be generated using other methods, such as SLAM (Simultaneous Localization and Mapping). The electromagnetic wave transmitting and receiving device may then add location information of the electromagnetic wave transmitting and receiving device at the time the reflected wave was received to the above-mentioned reflected wave information.

[0050] The location information may be information separate from the reflected wave information. In this case, the location information is time-series information of the location of the electromagnetic wave transmitting and receiving device. This time-series information includes a combination of a date and time and the location of the electromagnetic wave transmitting and receiving device at that date and time.

[0051] The reflected wave information generated by the electromagnetic wave transmitting and receiving device is input to the information processing device 10 by any means. For example, the electromagnetic wave transmitting and receiving device and the information processing device 10 may be configured to be able to communicate with each other. The electromagnetic wave transmitting and receiving device may then transmit the generated reflected wave information to the information processing device 10 via the communication means. The transmission of the reflected wave information from the electromagnetic wave transmitting and receiving device to the information processing device 10 may be performed by real-time processing or batch processing.

[0052] Alternatively, the reflected wave information generated by the electromagnetic wave transmitting and receiving device may be stored in any storage device. The storage device may be provided within the electromagnetic wave transmitting and receiving device, or may be provided in an external device configured to be able to communicate with the electromagnetic wave transmitting and receiving device. The reflected wave information stored in the storage device may then be input to the information processing device 10 at any timing and by any means.

[0053] The position information can be input to the information processing device 10 in the same manner as the reflected wave information.

[0054] The "second reliability information" indicates an area of ​​the target area where a target object may exist. The above-mentioned first reliability information is generated by processing the visible image based on the search criteria and a language model. In contrast, the second reliability information is generated based on reflected wave information indicating reflected waves of electromagnetic waves.

[0055] FIG. 4 shows an example of the second reliability information. The illustrated second reliability information is obtained by dividing a two-dimensional image showing the state of reflected waves from a target region into multiple sub-regions, and indicating the reliability of the presence of a target object for each sub-region. The higher the reliability of a sub-region, the more likely it is that a target object is present. One sub-region may correspond to one pixel. Alternatively, one sub-region may correspond to multiple pixels. The reflected wave processing unit 13 generates such second reliability information by processing the reflected wave information. The process of generating the second reliability information will be described below.

[0056] First, the reflected wave processing unit 13 generates three-dimensional information by processing the reflected wave information. Specifically, the reflected wave information includes a time-series signal indicating the intensity of the reflected wave and the position of the electromagnetic wave transmitting / receiving device. For example, the reflected wave processing unit 13 calculates the distance from the electromagnetic wave receiving unit to the reflection point from which the reflected wave originates by performing a Fast Fourier Transform (FFT) multiple times on the reflected wave constituting this time-series signal. If there are multiple electromagnetic wave receiving units, the reflected wave processing unit 13 performs this processing for each electromagnetic wave receiving unit. The reflected wave processing unit 13 then integrates multiple distances based on reflected waves measured by different electromagnetic wave receiving units at the same time to calculate an estimate of the intensity of the reflected wave for at least one first point included in the three-dimensional space corresponding to the target area. The reflected wave processing unit 13 then performs this processing on the reflected waves measured at multiple times to calculate estimates of the intensity of the reflected wave for each of the multiple first points, which are then used as three-dimensional information. This estimated value can be regarded as a value indicating the possibility that a target object exists at the first point. Hereinafter, this value will be referred to as a first value. However, the method of generating the three-dimensional information, for example, the method of generating the first value, is not limited to this example.

[0057] Next, the reflected wave processing unit 13 generates two-dimensional information by projecting the three-dimensional information onto a predetermined plane. Hereinafter, this predetermined plane will be referred to as the projection plane. The angle that the projection plane makes with the ground surface of the target area is preferably 10° or less. In other words, the projection plane is preferably parallel to the ground surface of the target area.

[0058] FIG. 5 is a diagram illustrating an example of processing performed by the reflected wave processing unit 13. The reflected wave processing unit 13 identifies multiple first points corresponding to a second point. For example, the reflected wave processing unit 13 determines multiple first points that overlap with a second point when viewed from a direction perpendicular to the projection surface as first points corresponding to the second point. Next, the reflected wave processing unit 13 identifies a first value corresponding to each of the identified multiple first points and uses the first value to generate a second value indicating the possibility that the target object is present at the second point. The second value may be a statistical value (maximum, minimum, average, mode, median, etc.) of the multiple first values. Alternatively, the second value may be the first value corresponding to the first point closest to the target area among the first points whose first values ​​exceed a reference value. The reflected wave processing unit 13 then determines the second value for each second point as two-dimensional information. In other words, the two-dimensional information can be considered as black and white image data (two-dimensional image). In this embodiment, one second point corresponds to one child region, but a collection of multiple second points may also correspond to one child region. When a collection of multiple second points corresponds to one child region, the statistical values ​​(maximum, minimum, average, mode, median, etc.) of multiple second values ​​corresponding to the multiple second points respectively become the second values ​​of the child region.

[0059] The reflected wave processing unit 13 then generates second reliability information based on the two-dimensional information. For example, measurements or the like are performed in advance to generate a probability distribution of the intensity of the reflected wave from the target object. The reflected wave processing unit 13 can then calculate the reliability of each second point based on the probability distribution and the second values ​​of each of the multiple second points in the two-dimensional information. The higher the probability of the second value indicated by the probability distribution, the higher the reliability.

[0060] It is also possible to set multiple conditions in advance based on the geology of the target area, the weather when the reflected waves were generated, the material components of the target object, etc., and generate the above-mentioned probability distribution for each setting. The reflected wave processing unit 13 may then generate second reliability information based on a probability distribution corresponding to the geology of the target area, the weather when the reflected waves were generated, the material components of the target object, etc. If the geology of the target area, the weather when the reflected waves were generated, the material components of the target object, etc., satisfy multiple conditions, the reflected wave processing unit 13 may generate multiple pieces of second reliability information based on the probability distribution corresponding to each condition. The reflected wave processing unit 13 may then integrate the multiple pieces of second reliability information to generate the second reliability information. An example of an integration method is a method of calculating a reliability statistical value (average, maximum, minimum, mode, median, etc.) for each second point, but this is not limited to this.

[0061] As another example, the reflected wave processing unit 13 may generate second reliability information in which the reliability increases as the second value increases. In this example, the reflected wave processing unit 13 calculates the reliability for each second point using a "rule for calculating reliability from the second value" that has been generated in advance. The rule is realized using a function, a table, or the like.

[0062] As another example, the reflected wave processing unit 13 may detect a cluster representing the shape of the target object from among a cluster of second points. The detection can be achieved using techniques such as pattern matching, feature matching, and a classifier generated by machine learning. Then, the reliability of the second points belonging to the detected cluster may be relatively increased.

[0063] As described above, the reflected wave processing unit 13 processes the reflected wave information to generate three-dimensional information indicating the possibility that a target object exists for a plurality of first points included in a three-dimensional space corresponding to a target region. The reflected wave processing unit 13 then projects the three-dimensional information onto a predetermined plane to generate two-dimensional information (two-dimensional images) indicating the possibility that a target object exists for each of a plurality of second points included in the plane. The reflected wave processing unit 13 then generates second reliability information based on the two-dimensional information.

[0064] As a modified example, the reflected wave processing unit 13 may perform processing to detect an area where the above-mentioned target object may exist, using at least one of geological information of the target area and weather information at the time the reflected wave was generated. For example, the geology of a specific area within the target area may be more likely to generate reflected waves. Also, depending on the weather, water or snow may accumulate on the surface of the target area, affecting the reflected wave. The reflected wave processing unit 13 can reflect this influence in the above-mentioned detection processing.

[0065] For example, the reflected wave processing unit 13 may generate three-dimensional information or two-dimensional information by multiplying the first value or the second value by a parameter corresponding to the geology of the location. This parameter is set in advance. The reflected wave processing unit 13 may also generate three-dimensional information or two-dimensional information by multiplying the first value or the second value by a parameter corresponding to the weather at the time of measurement. This parameter is also set in advance.

[0066] The geological information and weather information may be input to the information processing device 10 by a user of the information processing device 10, or may be acquired by the information processing device 10 from a database in which the information processing device 10 stores the information.

[0067] Returning to FIG. 1, the estimation unit 14 estimates an area in the target area where the target object exists, based on the first reliability information and the second reliability information.

[0068] In one example, the estimation unit 14 can estimate a child region in which the reliability indicated by the first reliability information is equal to or greater than a first threshold and the reliability indicated by the second reliability information is equal to or greater than a second threshold as a region in which a target object exists.

[0069] Alternatively, the estimation unit 14 may calculate an integrated score of the reliability indicated by the first reliability information and the reliability indicated by the second reliability information for each sub-region. Then, the estimation unit 14 may estimate a sub-region having an integrated score equal to or greater than a threshold as a region in which a target object exists. The calculation of the integrated score is realized using a function, a table, or the like.

[0070] For example, the estimation unit 14 may represent the first reliability information as a reliability map φ1 and the second reliability information as a reliability map φ2 as shown in FIG. 4, and set a linear or nonlinear function F(φ1, φ2) related to the reliability maps φ1 and φ2. The estimation unit 14 may then estimate a sub-region where the value of this function F is equal to or greater than a threshold as a region where a target object exists. The function F may be learned in advance by machine learning or the like. The function F may be learned as a regression function using, for example, a neural network or the like.

[0071] Incidentally, when integrating the reliability indicated by the first reliability information and the reliability indicated by the second reliability information for each child region as described above, it is necessary to identify the correspondence between the child regions of the first reliability information and the child regions of the second reliability information. Corresponding child regions are regions related to the same point within the target region. There are various means for realizing this identification, and one example will be described below.

[0072] "First Identification Example" In one example, the user may input to specify the correspondence. One example of the input is to use the first reliability information and the second reliability information shown in FIG. 4 . In this example, the first reliability information and the second reliability information shown in FIG. 4 are displayed on the screen. The user shifts the position of at least one of the first reliability information and the second reliability information to make the corresponding child regions overlap. The user can identify the corresponding child regions based on the shape, size, etc. of the clusters of child regions whose reliability indicated in the first reliability information and the second reliability information is equal to or greater than a predetermined value. The estimation unit 14 then identifies the overlapping child regions as corresponding child regions.

[0073] As another example of input, the "visible image" and "two-dimensional information (two-dimensional image) generated based on reflected wave information" in FIG. 4 may be displayed on the screen. Then, the user may shift the position of at least one of the visible image and the two-dimensional image so that corresponding points overlap with each other. Then, the estimation unit 14 may identify the overlapping child regions as corresponding child regions.

[0074] "Second Specific Example" The visible image processing unit 12 may transform the visible image into the same coordinate system as the two-dimensional image generated based on the reflected wave information, based on the positions of the visible camera and the electromagnetic wave transmitting and receiving device. If the visible camera and the electromagnetic wave transmitting and receiving device are mounted on the same measurement device, this transformation can be achieved based on the distance, orientation, etc. between the visible camera and the electromagnetic wave transmitting and receiving device. Transformation rules for transforming the visible image into a predetermined coordinate system based on the position information are prepared in advance. The visible image processing unit 12 performs the above-mentioned transformation of the visible image using the transformation rules. Examples of image transformation methods include, but are not limited to, image transformations such as affine transformation and homography transformation. The viewpoint of the transformed visible image becomes the same as the viewpoint of the two-dimensional image generated based on the reflected wave information. As a result, a child region at a predetermined position (e.g., center) in the visible image and a child region at the same predetermined position (e.g., center) in the two-dimensional image generated based on the reflected wave information correspond to each other.

[0075] Next, an example of the flow of processing executed by the information processing device 10 will be described with reference to the flowchart of FIG.

[0076] First, the information processing device 10 acquires search criteria that express the target object in text (S10). Then, as shown in FIG. 4, the information processing device 10 generates first reliability information by processing a visible image of the target area based on the search criteria and a language model (S11). The first reliability information indicates an area of ​​the target area where the target object may exist. Note that, as described in the second identification example above, the information processing device 10 may transform the visible image using a transformation rule prepared in advance, and then process the transformed visible image to generate the first reliability information.

[0077] 4, the information processing device 10 generates second reliability information by processing reflected wave information indicating reflected waves of the electromagnetic waves irradiated to the target area (S12). The second reliability information indicates an area of ​​the target area where a target object may exist.

[0078] Then, the information processing device 10 estimates an area in the target area where the target object exists based on the first reliability information and the second reliability information (S13). The information processing device 10 integrates the first reliability information and the second reliability information as shown in Fig. 4, and can estimate an area where the target object exists based on the integration result.

[0079] The information processing device 10 can output the estimation result. For example, the information processing device 10 may output an integration result as shown in FIG. 4 . Alternatively, the information processing device 10 may output an image in which the integration result as shown in FIG. 4 is superimposed on a visible image as shown in FIG. 4 . Alternatively, the information processing device 10 may output an image in which information indicating a region in which the target object is estimated to exist (a region in which the child regions are gathered) is superimposed on a visible image as shown in FIG. 4 . The information indicating the region in which the target object is estimated to exist (a region in which the child regions are gathered) may be a rectangular mark surrounding the region, or may be other information.

[0080] "Effects" The information processing device 10 estimates an area where a target object exists based on first reliability information generated by processing a visible image with a language model and second reliability information generated by processing reflected wave information indicating reflected waves of electromagnetic waves. By performing processing using a language model, the information processing device 10 is able to robustly detect new environments and new objects. Furthermore, by performing processing using a language model, the information processing device 10 can specify positive examples and negative examples to detect a target object with high accuracy. Furthermore, by performing processing using reflected wave information, the information processing device 10 can detect an object that is partially hidden, for example, underground and only partially visible in the visible image. The information processing device 10, which estimates an area where a target object exists based on both the first reliability information and the second reliability information, can accurately detect a target object.

[0081] The information processing device 10 can be used, for example, to search for metal objects (such as cans), diagnose infrastructure (diagnose structures including gas pipes and reinforcing bars), etc. Note that the use of the information processing device 10 is not limited to these.

[0082] Third Embodiment First, the differences between the second and third embodiments will be described with reference to FIGS.

[0083] In the second embodiment, the information processing device 10 receives input of one pattern of search conditions and generates one piece of first reliability information based on the received search conditions, as shown in Fig. 6. Then, the information processing device 10 estimates an area in which a target object exists based on the one piece of first reliability information and the second reliability information.

[0084] 7 , when the information processing device 10 receives input of one pattern of search conditions, it generates multiple patterns of search conditions based on the input. Then, the information processing device 10 generates multiple pieces of first reliability information based on each of the multiple patterns of search conditions. Then, the information processing device 10 estimates an area in which a target object exists based on the multiple pieces of first reliability information and the second reliability information.

[0085] In another example of the third embodiment, when the information processing device 10 receives input of a plurality of patterns of search conditions, the information processing device 10 generates a plurality of pieces of first reliability information based on each of the plurality of patterns of search conditions, as shown in Fig. 8. Then, the information processing device 10 estimates an area in which the target object exists based on the plurality of pieces of first reliability information and the second reliability information.

[0086] In another example of the third embodiment, the information processing device 10, although not shown, receives input of a plurality of patterns of search conditions and generates a plurality of patterns of search conditions from each of the plurality of patterns of search conditions.The information processing device 10 then generates a plurality of pieces of first reliability information based on each of the plurality of patterns of search conditions obtained.The information processing device 10 then estimates the area in which the target object exists based on the plurality of pieces of first reliability information and the second reliability information.This will be described in detail below.

[0087] "Example of FIG. 7" In this example, the search condition acquisition unit 11 accepts input of one pattern of search conditions. Then, the search condition acquisition unit 11 generates multiple patterns of search conditions from that one pattern of search conditions. A search condition of one pattern is a representation of a target object in one pattern. For example, "blue can" is one pattern of search conditions.

[0088] The search condition acquisition unit 11 generates search conditions that express the target object using other methods from this one pattern of search conditions. Other patterns of search conditions for "blue can" include, but are not limited to, "blue metal lump" and "blue cylindrical object." For example, dictionary data that paraphrases one word into multiple expressions may be generated in advance and registered in the information processing device 10 or an external device accessible from the information processing device 10. Then, the search condition acquisition unit 11 may generate multiple patterns of search conditions based on the dictionary data.

[0089] The visible image processing unit 12 performs object detection and object recognition based on a language model, based on the input search conditions and the search conditions generated by the search condition acquisition unit 11. Then, the visible image processing unit 12 generates first reliability information corresponding to each of the search conditions.

[0090] The estimation unit 14 estimates a region of the target region where the target object exists based on multiple pieces of first reliability information and second reliability information. As an example, the estimation unit 14 may integrate multiple pieces of first reliability information to generate one piece of first reliability information. An example of an integration method is a method of calculating a reliability statistical value (average, maximum, minimum, mode, median, etc.) for each child region, but this is not limiting. Then, the estimation unit 14 may estimate a region of the target region where the target object exists based on the one piece of first reliability information and the second reliability information using a method similar to that of the second embodiment.

[0091] "Example of FIG. 8" In this example, the search condition acquisition unit 11 accepts input of multiple patterns of search conditions. That is, the user inputs multiple patterns of search conditions that express the target object using multiple methods. The multiple patterns of search conditions include, but are not limited to, "blue can," "blue metal lump," and "blue cylindrical object."

[0092] Other configurations of the information processing device 10 in the example of FIG. 8 are similar to those in the example of FIG.

[0093] An example in which a plurality of patterns of search conditions are input and a plurality of patterns of search conditions are further generated from each of them can be realized by a method similar to the examples of FIGS. 7 and 8 described above.

[0094] Other configurations of the information processing apparatus 10 of the third embodiment are similar to those of the information processing apparatus 10 of the first and second embodiments.

[0095] The information processing device 10 of the third embodiment achieves the same effects as the information processing device 10 of the first and second embodiments. Furthermore, the information processing device 10 of the third embodiment can perform processing using a language model with search conditions of multiple patterns that represent a target object, and can integrate the results to detect the target object. This information processing device 10 can detect a target object with higher accuracy.

[0096] <Modifications> Modifications applicable to the first to third embodiments will be described.

[0097] After performing correction processing on the visible image, the visible image processing unit 12 may perform object recognition or object detection using a language model on the corrected visible image. As an example, the visible image processing unit 12 may perform correction to adjust the brightness of the visible image. The visible image processing unit 12 may adjust the brightness of the visible image using, for example, the technology disclosed in the following document. The visible image processing unit 12 may perform this correction when the brightness of the visible image satisfies a predetermined condition (brightness is equal to or less than a threshold): "Shibata, Takashi, Masayuki Tanaka, and Masatoshi Okutomi. "Gradient-domain image reconstruction framework with intensity-range and base-structure constraints." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016." "Tanaka, Masayuki, Takashi Shibata, and Masatoshi Okutomi. "Gradient-based low-light image enhancement." 2019 IEEE International Conference on Consumer Electronics (ICCE). IEEE, 2019."

[0098] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0099] In addition, in the flowcharts used in the above explanation, multiple steps (processes) are described in order. However, the order of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not cause any problems in terms of the content.

[0100] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes: 1. An information processing device comprising: a search condition acquisition means for acquiring search conditions that express a target object in text; a visible image processing means for generating first reliability information that indicates an area within the target area where the target object may be present by processing a visible image of a target area based on the search conditions and a language model; a reflected wave processing means for generating second reliability information that indicates an area within the target area where the target object may be present by processing reflected wave information that indicates reflected waves of electromagnetic waves irradiated onto the target area; and an estimation means for estimating an area within the target area where the target object is present based on the first reliability information and the second reliability information. 2. The information processing device described in 1, wherein the search conditions include: text that expresses the target object; and text that expresses an object to be removed from search results. 3. The information processing device described in 1 or 2, wherein the first reliability information and the second reliability information indicate a reliability that the target object is present for each of a plurality of child areas within the target area. 4. The information processing device according to any one of 1 to 3, wherein the search condition acquisition means acquires a plurality of patterns of the search conditions, the visible image processing means generates a plurality of pieces of first reliability information by processing the visible image based on each of the plurality of patterns of the search conditions, and the estimation means estimates a region of the target region in which the target object is present based on the plurality of pieces of first reliability information and the second reliability information. 5. The information processing device according to 4, wherein the search condition acquisition means accepts input of a plurality of patterns of the search conditions from a user. 6. The information processing device according to 4, wherein the search condition acquisition means generates a plurality of patterns of the search conditions from one pattern of the search conditions. 7. The information processing device according to any one of 1 to 6, wherein the frequency of the electromagnetic waves is 0.3 GHz or more and 300 GHz or less.8. The information processing device according to any one of 1 to 7, wherein the reflected wave processing means processes the reflected wave information to generate three-dimensional information indicating the possibility that the target object exists for a plurality of first points included in a three-dimensional space corresponding to the target area, projects the three-dimensional information onto a predetermined plane to generate two-dimensional information indicating the possibility that the target object exists for each of a plurality of second points included in the plane, and generates the second reliability information based on the two-dimensional information. 9. An information processing method in which one or more computers acquire search conditions that express the target object in text, process visible images of a captured target area based on the search conditions and a language model to generate first reliability information indicating an area in the target area where the target object may exist, process reflected wave information that indicates reflected waves of electromagnetic waves irradiated to the target area to generate second reliability information indicating an area in the target area where the target object may exist, and estimate the area in the target area where the target object exists based on the first reliability information and the second reliability information. A program that causes a computer to function as: a search condition acquisition means that acquires search conditions that express a target object in text; a visible image processing means that processes a visible image of a target area based on the search conditions and a language model to generate first reliability information that indicates an area within the target area where the target object may be present; a reflected wave processing means that processes reflected wave information that indicates a reflected wave of an electromagnetic wave irradiated to the target area to generate second reliability information that indicates an area within the target area where the target object may be present; and an estimation means that estimates an area within the target area where the target object is present based on the first reliability information and the second reliability information.

[0101] Some or all of Supplements 2 to 8 that are dependent on the information processing device of Supplement 1 described above may also be dependent on the information processing method of Supplement 9 and the program of Supplement 10 in the same dependent relationship as Supplement 1 and Supplements 2 to 8. Furthermore, within the scope of each of the above-mentioned embodiments, some or all of the configurations described as Supplements can be realized in various hardware, software, various recording means for recording software, or systems.

[0102] This application claims priority based on Japanese Patent Application No. 2023-126859, filed on August 3, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0103] REFERENCE SIGNS LIST 10 Information processing device 11 Search condition acquisition unit 12 Visible image processing unit 13 Reflected wave processing unit 14 Estimation unit 1A Processor 2A Memory 3A Input / output I / F 4A Peripheral circuit 5A Bus

Claims

1. A search condition acquisition means that acquires search conditions that express the target object in text, A visible image processing means generates first confidence information indicating the region within the target area in which the target object may exist by processing a visible image of the target area based on the search conditions and language model, A reflected wave processing means generates a second reliability information indicating a region within the target region in which the target object may exist, by processing reflected wave information indicating reflected waves of electromagnetic waves irradiated onto the target region. Estimation means for estimating the region within the target area in which the target object exists, based on the first reliability information and the second reliability information, An information processing device having

2. The aforementioned search criteria are: Text describing the aforementioned object, Text describing the object you want to remove from the search results, The information processing apparatus according to claim 1, including the following:

3. The information processing apparatus according to claim 1 or 2, wherein the first reliability information and the second reliability information indicate the reliability of the existence of the target object for each of the plurality of sub-regions within the target region.

4. The search condition acquisition means acquires the search conditions in multiple patterns, The visible image processing means generates a plurality of first confidence information by processing the visible image based on each of the plurality of search conditions. The information processing apparatus according to claim 1 or 2, wherein the estimation means estimates the region in the target region in which the target object exists based on a plurality of first reliability information and second reliability information.

5. The information processing apparatus according to claim 4, wherein the search condition acquisition means receives input of multiple patterns of the search conditions from a user.

6. The information processing apparatus according to claim 4, wherein the search condition acquisition means generates a plurality of search condition patterns from one pattern of the search condition.

7. The information processing apparatus according to claim 1 or 2, wherein the frequency of the electromagnetic wave is 0.3 GHz or more and 300 GHz or less.

8. The reflected wave processing means is By processing the reflected wave information, three-dimensional information is generated indicating the possibility of the existence of the target object for a plurality of first points included in the three-dimensional space corresponding to the target region. By projecting the three-dimensional information onto a predetermined plane, two-dimensional information is generated indicating the possibility of the target object existing at each of a plurality of second points contained within the plane. The information processing apparatus according to claim 1 or 2, which generates the second reliability information based on the two-dimensional information.

9. One or more computers, Obtain search criteria that describe the target object in text, By processing the visible image captured from the target area based on the search conditions and language model, a first confidence level information is generated that indicates the area within the target area where the target object may exist. By processing reflected wave information indicating the reflected waves of electromagnetic waves irradiated onto the target region, a second reliability information is generated that indicates the region within the target region where the target object may exist. An information processing method for estimating the region within the target region in which the target object exists, based on the first reliability information and the second reliability information.

10. Computers, A search condition acquisition means that acquires search conditions that express the target object in text. Visible image processing means that processes a visible image of the target area based on the search conditions and language model to generate first confidence information indicating the area within the target area where the target object may exist. A reflected wave processing means generates a second reliability information indicating a region within the target area in which the target object may be present, by processing reflected wave information indicating reflected waves of electromagnetic waves irradiated onto the target area. Estimation means for estimating the region within the target area in which the target object exists, based on the first reliability information and the second reliability information. A program that makes it function as such.