Object detection device, object detection method, and program

Abstract

The present invention provides an object detection device comprising: an image generation unit that generates a reflected wave image on the basis of received signals of electromagnetic waves reflected by objects; and a likelihood map generation unit that generates, on the basis of the intensity of the received signals indicated in the reflected wave image, a first likelihood map indicating the likelihood of each pixel in the reflected wave image corresponding to an irregularly shaped object occluding a background object and having differing electromagnetic wave reflectivity from the background object, and generates, on the basis of shapes of the objects shown in the reflected wave image, a second likelihood map indicating the likelihood of each pixel in the reflected wave image corresponding to a predefined fixed-shape object.

Description

Object detection device, object detection method, and recording medium

[0001] The present invention relates to an object detection device, an object detection method, and a program.

[0002] Techniques related to the present invention are disclosed in Patent Documents 1 to 4. The techniques disclosed in Patent Documents 1 to 4 relate to inspection of belongings using electromagnetic waves.

[0003] The technology disclosed in Patent Document 1 uses data on the intensity and depth along the surface of the object to check whether a target person is carrying man-made objects that are suspected of being contraband or a security threat (such as explosives).

[0004] US Patent No. 5,949,693 discloses that objects may be visible in X-ray images of the human body due to differences in X-ray reflectivity between objects and human tissue.

[0005] The technology disclosed in Patent Document 3 identifies the type of object contained in a detection target from the difference between the radiant energy signal emitted by the detection target and the radiant energy signal of the detection target surrounding the non-detection target. The technology disclosed in Patent Document 3 can inspect a wide range of substances, from powders to solids such as metals.

[0006] Patent Document 4 discloses that the reflectance of electromagnetic waves reflected by the human body differs from that of electromagnetic waves reflected by an object concealed in clothing, and that the concealed object can be detected based on this.

[0007] JP 2007-517275 A JP 2010-530977 A JP 2017-40593 A JP 2021-32778 A

[0008] In a personal possession search, there are cases where both fixed-shaped objects and irregular-shaped objects are to be detected. Fixed-shaped objects include, for example, guns and blades. Irregular-shaped objects include, for example, liquids and powders. Therefore, there is a demand for technology that can accurately detect both fixed-shaped objects and irregular-shaped objects.

[0009] The techniques described in Patent Documents 1 to 4 can detect a wide range of objects. However, these techniques use the same method to detect both fixed-shape objects and irregular-shape objects. Unless a method suited to each object is used, detection accuracy will be poor.

[0010] In view of the above-mentioned problems, one example of the object of the present invention is to provide an object detection device, an object detection method, and a program that solve the problem of accurately detecting both objects with fixed shapes and objects with irregular shapes.

[0011] According to one aspect of the present invention, there is provided an object detection device comprising: an image generation means for generating a reflected wave image based on a received signal of electromagnetic waves reflected by an object; and a likelihood map generation means for generating, based on the intensity of the received signal shown in the reflected wave image, a first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that occludes a background object and has a reflectivity of electromagnetic waves different from that of the background object, and for generating, based on the shape of the object shown in the reflected wave image, a second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape.

[0012] According to one aspect of the present invention, an object detection method is provided in which one or more computers generate a reflected wave image based on received signals of electromagnetic waves reflected by an object, generate a first likelihood map based on the intensity of the received signals shown in the reflected wave image, indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that occludes a background object and has a reflectivity of electromagnetic waves different from that of the background object, and generate a second likelihood map based on the shape of the object shown in the reflected wave image, indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape.

[0013] According to one aspect of the present invention, there is provided a program that causes a computer to function as: an image generation means that generates a reflected wave image based on a received signal of electromagnetic waves reflected by an object; and a likelihood map generation means that generates a first likelihood map based on the intensity of the received signal shown in the reflected wave image, indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that occludes a background object and has a reflectivity of electromagnetic waves different from that of the background object; and that generates a second likelihood map based on the shape of the object shown in the reflected wave image, indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape.

[0014] According to one aspect of the present invention, an object detection device, an object detection method, and a program are realized that solve the problem of accurately detecting both objects with fixed shapes and objects with irregular shapes.

[0015] The above and other objects, features, and advantages will become more apparent from the following description of the preferred embodiments and the accompanying drawings.

[0016] FIG. 1 is a diagram showing an example of a functional block diagram of an object detection device. FIG. 2 is a diagram for explaining features of a learning model used by the object detection device. FIG. 3 is a diagram for explaining a comparative example. FIG. 4 is a diagram showing an example of a hardware configuration of an object detection device. FIG. 5 is a diagram showing an example of a functional block diagram of an object detection device. FIG. 6 is a diagram showing an example of a reflected wave image generated by the object detection device. FIG. 7 is a diagram showing an example of a set of likelihood maps generated by the object detection device. FIG. 8 is a diagram showing another example of a set of likelihood maps generated by the object detection device. FIG. 9 is a diagram showing an example of a functional block diagram of a learning device. FIG. 10 is a flowchart showing an example of a processing flow of the object detection device. FIG. 11 is a diagram showing another example of a functional block diagram of the object detection device. FIG. 12 is a diagram showing an example of a first image and a second image generated by the object detection device. FIG. 13 is a diagram showing another example of the first image and the second image generated by the object detection device. FIG. 14 is a flowchart showing an example of a processing flow of the object detection device.

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0018] 1 is a functional block diagram showing an overview of an object detection device 10 according to a first embodiment. The object detection device 10 includes an image generation unit 11 and a likelihood map generation unit 12.

[0019] The image generating unit 11 generates a reflected wave image based on a received signal of an electromagnetic wave reflected by an object.

[0020] The likelihood map generator 12 generates a first likelihood map and a second likelihood map.

[0021] The first likelihood map indicates the likelihood that each pixel of the reflected wave image corresponds to an irregularly shaped object that occludes background objects and has a different reflectivity of electromagnetic waves than the background objects (hereinafter, sometimes simply referred to as an "irregularly shaped object"). That is, the first likelihood map indicates the likelihood that each pixel of the reflected wave image indicates the intensity of the reflected wave reflected by the irregularly shaped object. The likelihood map generator 12 generates the first likelihood map based on the intensity of the received signal indicated in the reflected wave image, using the difference between the reflectivity of the electromagnetic waves of the background objects and the reflectivity of the electromagnetic waves of the irregularly shaped object.

[0022] The second likelihood map indicates the likelihood that each pixel in the reflected wave image corresponds to an object of a predefined fixed shape. That is, the second likelihood map indicates the likelihood that each pixel in the reflected wave image indicates the intensity of a reflected wave reflected by an object of a fixed shape. The likelihood map generator 12 generates the second likelihood map using the shape of the object shown in the reflected wave image.

[0023] According to the object detection device 10 of the first embodiment, an object of an irregular shape is detected based on the strength of the received signal shown in the reflected wave image, utilizing the difference in the reflectance of the electromagnetic waves between the background objects and the object of an irregular shape. Furthermore, an object of a fixed shape is detected using the shape of the object shown in the reflected wave image. According to the object detection device 10 of the first embodiment, the problem of accurately detecting both an object of a fixed shape and an object of an irregular shape is solved.

[0024] <Second embodiment> "Overview" The object detection device 10 of the second embodiment is a specific implementation of the configuration of the object detection device 10 of the first embodiment. The object detection device 10 of the second embodiment has the same configuration as the object detection device 10 of the first embodiment, and achieves the same effects. In addition, the object detection device 10 of the second embodiment has the following features.

[0025] As described in the first embodiment, the object detection device 10 generates a first likelihood map indicating the likelihood that each pixel in the reflected wave image corresponds to an object of an irregular shape, and a second likelihood map indicating the likelihood that each pixel in the reflected wave image corresponds to an object of a fixed shape.

[0026] The object detection device 10 then generates both the first likelihood map and the second likelihood map using a previously generated learning model. As shown in Fig. 2, the learning model receives an input of a reflected wave image. The learning model then separately generates and outputs both the first likelihood map and the second likelihood map in response to the input of the reflected wave image.

[0027] FIG. 3 shows the configuration of a comparative example. The comparative example uses two learning models: a first learning model that generates a first likelihood map in response to an input of a reflected wave image, and a second learning model that generates a second likelihood map in response to an input of a reflected wave image. That is, the comparative example uses two learning models to generate the first likelihood map and the second likelihood map. The configuration of this comparative example has problems such as increased computer resources. The object detection device 10 of the second embodiment can alleviate these problems.

[0028] The configuration of the object detection device 10 according to the second embodiment will be described in detail below.

[0029] "Hardware Configuration" Next, an example of the hardware configuration of the object detection device 10 will be described. Each functional unit of the object detection device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various variations in the realization method and device. The software includes programs that are pre-stored in the device before shipping, and programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet.

[0030] FIG. 4 is a block diagram illustrating an example of the hardware configuration of the object detection device 10. As shown in FIG. 4, the object detection device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The object detection device 10 does not necessarily have to have the peripheral circuit 4A. Note that the object detection device 10 may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices can have the above hardware configuration.

[0031] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to mutually transmit and receive data. The processor 1A is, for example, a processing unit such as a CPU or a graphics processing unit (GPU). The memory 2A is, for example, a random access memory (RAM) or a read-only memory (ROM). The input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. The input / output interface 3A also includes an interface for connecting to a communication network such as the Internet. Examples of input devices include a keyboard, mouse, microphone, physical buttons, touch panel, etc. Examples of output devices include a display, speaker, printer, mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0032] "Functional Configuration" Next, the functional configuration of the object detection device 10 according to the second embodiment will be described in detail. Fig. 5 shows an example of a functional block diagram of the object detection device 10. As shown in the figure, the object detection device 10 includes an image generation unit 11, a likelihood map generation unit 12, an output unit 15, a learning model generation unit 16, and a learning model storage unit 17.

[0033] The image generating unit 11 generates a reflected wave image based on a received signal of an electromagnetic wave reflected by an object.

[0034] In the "reflected wave image," each pixel indicates the intensity (intensity of the received signal) of the electromagnetic wave (reflected wave) reflected by an object. FIG. 6 shows an example of a reflected wave image. The illustrated reflected wave image shows a human body. The generation of a reflected wave image based on the received electromagnetic wave signal can be achieved using any known technology.

[0035] The "object" refers to a human body, possessions carried by the human body, etc. The object detection device 10 is used, for example, in a possession search. In this case, the object refers to the human body of a person who is the subject of a possession search and possessions carried by that person.

[0036] The "electromagnetic wave reception signal" is generated by an electromagnetic wave transmission / reception device. The object detection device 10 may be equipped with an electromagnetic wave transmission / reception device. Alternatively, the electromagnetic wave transmission / reception device may be a separate device that is physically and / or logically separated from the object detection device 10. The electromagnetic wave reception signal generated by the electromagnetic wave transmission / reception device may be input to the object detection device 10 by any means.

[0037] The electromagnetic wave transmitting and receiving device controls a transmitting antenna that radiates electromagnetic waves such as millimeter waves and a receiving antenna that receives reflected waves of the electromagnetic waves radiated from the transmitting antenna. The electromagnetic wave transmitting and receiving device controls, for example, the radiation of the electromagnetic waves from the transmitting antenna, specifically the radiation timing, etc.

[0038] The electromagnetic waves emitted by the transmitting antenna can be, for example, continuous waves (CW), frequency modulated continuous waves (FMCW), stepped frequency continuous waves (SFCW), etc. The receiving antenna measures the complex amplitude of the received waves (a complex number representing the amplitude and phase shift from the transmitted wave) for each frequency, and the measurement results are used as radar signals. The radar signal can be expressed as S(n, m, f) using the transmitting antenna number n, receiving antenna number m, and frequency f as arguments.

[0039] The antenna configuration is not particularly limited and any configuration can be used, for example, an antenna panel in which multiple transmitting antennas and multiple receiving antennas are arranged may be used.

[0040] The multiple antennas are installed in positions and orientations that irradiate electromagnetic waves into a target space and receive waves reflected by objects located within the target space.

[0041] The target space is a space where a predetermined inspection, such as a possessions inspection, is carried out. The above-mentioned antenna irradiates electromagnetic waves onto the target space including the target object and receives the reflected waves. Various inspections are then carried out based on the radar signals of the reflected waves. The target space is provided, for example, in a passageway through which the target object passes. Then, while the target object is moving within the target space, electromagnetic waves are irradiated and the reflected waves are received. In this way, in one example of the second embodiment, a walk-through type inspection is realized. Alternatively, the target space may be a space where the target object will stop for a certain period of time for inspection.

[0042] Returning to FIG. 5, the likelihood map generator 12 generates a first likelihood map and a second likelihood map.

[0043] The "first likelihood map" indicates the likelihood that each pixel of the reflected wave image corresponds to an irregularly shaped object that occludes a background object and has a different reflectivity of electromagnetic waves from the background object. That is, the first likelihood map indicates the likelihood that each pixel of the reflected wave image indicates the intensity of the reflected wave reflected by the irregularly shaped object.

[0044] The likelihood map generator 12 generates a first likelihood map based on the intensity of the received signal shown in the reflected wave image, utilizing the difference between the reflectance of the electromagnetic wave of the background object and the reflectance of the electromagnetic wave of the irregularly shaped object. The irregularly shaped object to be detected overlaps with the background object in the reflected wave image, occluding it. The reflectance of the electromagnetic wave of the irregularly shaped object differs from the reflectance of the electromagnetic wave of the background object. Due to this characteristic, the intensity of the received signal of a portion of the background object (a portion occluded by the irregularly shaped object) in the reflected wave image is lower than that of other portions (a portion not occluded by the irregularly shaped object). Based on this phenomenon, the likelihood that each pixel in the reflected wave image corresponds to an irregularly shaped object can be calculated.

[0045] The "background object" is, for example, a human body. The "object of an irregular shape" includes, for example, a liquid or powder. The liquid is a liquid that is prohibited from being possessed or brought into a facility. The powder is a powder-like substance that is prohibited from being possessed or brought into a facility, such as narcotics.

[0046] The "second likelihood map" indicates the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape. In other words, the second likelihood map indicates the likelihood that each pixel of the reflected wave image indicates the intensity of a reflected wave reflected by an object of a predefined fixed shape.

[0047] The likelihood map generator 12 generates a second likelihood map based on the shape of the object shown in the reflected wave image. A fixed-shape object has a fixed shape. Therefore, based on the shape of the object shown in the reflected wave image and the shape of the predefined fixed-shape object, it is possible to calculate the likelihood that each pixel in the reflected wave image corresponds to the predefined fixed-shape object.

[0048] The "fixed-shape object" includes a gun, a bladed weapon, or other weapon. A predetermined type of object is defined in advance as the fixed-shape object. There may be one or more fixed-shape objects defined in advance.

[0049] The likelihood map generator 12 generates both a first likelihood map and a second likelihood map separately in response to the input of a reflected wave image, and generates the first likelihood map and the second likelihood map based on the output learning model (see Figure 2).

[0050] When a reflected wave image is input to the learning model, multiple likelihood maps such as those shown in FIG. 7 are output. In the example of FIG. 7 , the learning model generates and outputs five likelihood maps in response to the input reflected wave image. Specifically, likelihood maps corresponding to each of "firearm," "sword," "blank (nothing)," "human body," and "object of irregular shape" are output. In FIG. 7 , the likelihood of the corresponding object type is displayed in shades of black and white, with areas with high likelihood being white and areas with low likelihood being black. In the example shown in FIG. 7 , two objects, "firearm" and "sword," are defined in advance as fixed-shape objects. Then, in response to the input reflected wave image, the learning model generates and outputs likelihood maps (second likelihood maps) corresponding to each of the two fixed-shape objects.

[0051] The likelihood map corresponding to "gun" indicates the likelihood that each pixel of the reflected wave image indicates the intensity of the reflected wave reflected by the gun. The likelihood map corresponding to "sword" indicates the likelihood that each pixel of the reflected wave image indicates the intensity of the reflected wave reflected by the sword. In the example of Figure 7, these fixed-shaped objects were not being held. Therefore, in both likelihood maps, the likelihood that each pixel of the reflected wave image indicates the intensity of the reflected wave reflected by the gun and the sword is low.

[0052] The likelihood map corresponding to "blank (nothing)" indicates the likelihood that there was nothing at the position of each pixel in the reflected wave image.

[0053] The likelihood map corresponding to the "human body" indicates the likelihood that each pixel of the reflected wave image indicates the intensity of the reflected wave reflected by the human body. The likelihood may be calculated based on, for example, the shape of the human body.

[0054] The likelihood map (first likelihood map) corresponding to the "irregularly shaped object" indicates the likelihood that each pixel in the reflected wave image indicates the intensity of the reflected wave reflected by the irregularly shaped object. In the example of FIG. 7, an irregularly shaped object was being held. Therefore, some pixels in the reflected wave image have a high likelihood of indicating the intensity of the reflected wave reflected by the irregularly shaped object.

[0055] Figure 8 shows another example of a likelihood map output from the learning model. As with Figure 7, Figure 8 also displays the likelihood of the corresponding object type in shades of black and white, with high likelihood areas in white and low likelihood areas in black. In this example, a firearm, an object of fixed shape, was being carried. Therefore, in the likelihood map corresponding to the firearm (second likelihood map), some pixels in the reflected wave image have a high likelihood indicating the intensity of the reflected wave reflected by the firearm. Note that in the example of Figure 8, neither a sword nor an object of irregular shape was being carried. Therefore, in both likelihood maps, the likelihood of each pixel in the reflected wave image indicating the intensity of the reflected wave reflected by the sword and the object of irregular shape is low.

[0056] Returning to Fig. 5, the learning model generation unit 16 generates the above-mentioned learning model and stores it in the learning model storage unit 17. The learning model generation unit 16 has the configuration of a learning device 20 described below. As shown in Fig. 9, the learning device 20 has a first learning data input unit 21, a second learning data input unit 22, and a learning unit 23. An example of the hardware configuration of the learning device 20 is shown in Fig. 4.

[0057] The first learning data input unit 21 inputs a first learning image and first correct answer data to the learning device 20 as learning data relating to an object of an irregular shape.

[0058] In the "first training image," each pixel represents the intensity of the reflected electromagnetic wave (the intensity of the received signal). The first training image represents the intensity of the electromagnetic wave reflected by an object of an irregular shape and the intensity of the electromagnetic wave reflected by a human body holding the object of an irregular shape. In the scene where the first training image is generated, electromagnetic waves are irradiated toward the human body holding the object of an irregular shape, and the reflected waves are received. Then, the first training image is generated based on the intensity of the received signal (the intensity of the reflected electromagnetic wave).

[0059] The "first correct answer data" indicates the region of the irregularly shaped object in the first training image. The first correct answer data may be generated manually by the user. That is, the user may visually recognize the first training image, identify the region of the irregularly shaped object in the first training image, and provide input specifying the identified region to the training device 20.

[0060] Alternatively, the first training data input unit 21 may generate the first supervised answer data from the first training images without the manual operation by the user as described above. An example of a process in which the first training data input unit 21 generates the first supervised answer data from the first training images will be described below.

[0061] - Example 1 of a process for generating first supervised answer data from a first training image - For example, the first training data input unit 21 detects, from the first training image, an area where a predetermined feature amount obtained from image intensity (strength of a received signal) satisfies a predetermined condition, as an area of ​​an object with an irregular shape. Then, the first training data input unit 21 uses the detection result (data indicating the detected area) as the first supervised answer data.

[0062] The "predetermined feature amount" is, for example, a dielectric constant. The first training data input unit 21 processes the first training image to generate a dielectric constant image in which the influence of position and the like has been corrected. The dielectric constant image indicates the dielectric constant corresponding to each pixel. The first training data input unit 21 then detects regions where the dielectric constant is equal to or less than a threshold as regions of irregularly shaped objects. In this method, regions of irregularly shaped objects are detected by utilizing the fact that the dielectric constant of the irregularly shaped objects is equal to or less than a threshold.

[0063] Alternatively, the predetermined feature may be the intensity of the received signal itself shown in the reflected wave image. In this case, the first learning data input unit 21 detects, as the region of the irregular-shaped object, the region in which the intensity of the received signal is within a predetermined numerical range (a value calculated based on the reflectance of the irregular-shaped object, etc.).

[0064] --Example 2 of a process for generating first ground truth data from a first learning image-- In this example, the electromagnetic wave transmitting and receiving device described above is used to irradiate an object with electromagnetic waves, and the reflected waves are received. The object is then photographed with a camera (e.g., a camera equipped with a sensor that detects visible light) to generate an image (hereinafter, "camera image"). The electromagnetic wave transmitting and receiving device and the camera are installed in approximately the same position and capture the object from approximately the same orientation.

[0065] The first training data input unit 21 identifies the position of an irregularly shaped object in a camera image generated by a camera. Then, the first training data input unit 21 identifies a position in the first training image that corresponds to the position of the irregularly shaped object identified in the camera image, and detects the identified position in the first training image as the region of the irregularly shaped object.

[0066] One method for identifying the position of an irregularly shaped object in a camera image is to use a marker. That is, when photographing a human body holding an irregularly shaped object, a marker that clearly indicates the position of the irregularly shaped object is captured in the camera image. The marker may be, for example, a sticker with a predetermined mark. In this case, the sticker is attached to the position of the irregularly shaped object. The detection of the marker in the camera image is achieved using conventional image analysis technology.

[0067] Next, a process for detecting an area of ​​an irregularly shaped object in the first learning image based on the position of the irregularly shaped object identified in the camera image will be described.

[0068] As described above, the camera image and the first training image are images of a human body holding an irregularly shaped object captured from approximately the same position and approximately the same direction. Therefore, the relative positional relationship between the human body and the irregularly shaped object shown in the camera image is equivalent to the relative positional relationship between the human body and the irregularly shaped object in the first training image. Based on this relationship, the first training data input unit 21 identifies a "position in the first training image" that corresponds to the "position of the irregularly shaped object identified in the camera image." The first training data input unit 21 then determines a predetermined region based on the identified position in the first training image as the region of the irregularly shaped object in the first training image. An example of a predetermined region based on the identified position in the first training image is a rectangular region centered on the identified position, but is not limited to this.

[0069] The second learning data input unit 22 inputs the second learning image and the second correct answer data to the learning device 20 as learning data relating to an object of a fixed shape.

[0070] In the "second training image," each pixel indicates the intensity of the reflected electromagnetic wave (the intensity of the received signal). The second training image indicates the intensity of the electromagnetic wave reflected by an object of a fixed shape and the intensity of the electromagnetic wave reflected by a human body holding the object of the fixed shape. In the scene where the second training image is generated, electromagnetic waves are irradiated toward a human body holding an object of a fixed shape, and the reflected waves are received. Then, the second training image is generated based on the intensity of the received signal (the intensity of the reflected electromagnetic wave).

[0071] The "second correct answer data" indicates the area of ​​the fixed-shape object in the second training image. The second correct answer data may be generated manually by the user. That is, the user may visually recognize the second training image, identify the area of ​​the fixed-shape object in the second training image, and provide input specifying the identified area to the training device 20.

[0072] Alternatively, the second training data input unit 22 may generate the second correct answer data from the second training images without the manual operation by the user as described above.

[0073] For example, the electromagnetic wave transmitting and receiving device described above is used to irradiate an object with electromagnetic waves, and the reflected waves are received. At the same time, the object is photographed with a camera (e.g., a camera equipped with a sensor that detects visible light) to generate a camera image. The electromagnetic wave transmitting and receiving device and the camera are installed in approximately the same position and capture the object from approximately the same direction.

[0074] The second training data input unit 22 identifies the position of an object of a fixed shape in a camera image generated by the camera, and then identifies a position in the second training image that corresponds to the position of the object of a fixed shape identified in the camera image, and detects the identified position in the second training image as the region of the object of a fixed shape.

[0075] One method for identifying the position of a fixed-shape object in a camera image is to use a marker. That is, when a human body holding a fixed-shape object is photographed, a marker that clearly indicates the position of the fixed-shape object is captured in the camera image. The marker may be, for example, a sticker with a predetermined mark. In this case, the sticker is attached to the position of the fixed-shape object. The detection of the marker in the camera image is achieved using conventional image analysis technology.

[0076] Next, a process for detecting the region of the fixed-shape object in the second learning image based on the position of the fixed-shape object identified in the camera image will be described.

[0077] As described above, the camera image and the second training image are images of a human body holding a fixed-shape object captured from approximately the same position and approximately the same direction. Therefore, the relative positional relationship between the human body and the fixed-shape object shown in the camera image is equivalent to the relative positional relationship between the human body and the fixed-shape object in the second training image. Based on this relationship, the second training data input unit 22 identifies a "position in the second training image" that corresponds to the "position of the fixed-shape object identified in the camera image." The second training data input unit 22 then determines a predetermined region based on the identified position in the second training image as the region of the fixed-shape object in the second training image. An example of a predetermined region based on the identified position in the second training image is a rectangular region centered on the identified position, but is not limited to this.

[0078] The learning unit 23 generates the above-mentioned learning model by learning based on the learning data relating to objects of irregular shapes input by the first learning data input unit 21 and the learning data relating to objects of fixed shapes input by the second learning data input unit 22. There are no particular limitations on the learning method.

[0079] Next, an example of the processing flow of the object detection device 10 will be described with reference to the flowchart of FIG.

[0080] First, the object detection device 10 generates a reflected wave image based on a received signal of an electromagnetic wave reflected by an object (S10).

[0081] Next, the object detection device 10 generates a first likelihood map and a second likelihood map based on the reflected wave image generated in S10 (S11). Specifically, the object detection device 10 generates a first likelihood map indicating the likelihood that each pixel in the reflected wave image corresponds to an object of an irregular shape based on the intensity of the received signal shown in the reflected wave image. The object detection device 10 also generates a second likelihood map indicating the likelihood that each pixel in the reflected wave image corresponds to an object of a predefined fixed shape based on the shape of the object shown in the reflected wave image.

[0082] Then, the object detection device 10 outputs the generated first likelihood map and second likelihood map (S12). For example, the object detection device 10 outputs a set of likelihood maps such as those shown in Fig. 7 or 8. The object detection device 10 outputs the maps via any output device such as a display, a projection device, a printer, or the like.

[0083] "Effects" The object detection device 10 of the second embodiment achieves the same effects as the object detection device 10 of the first embodiment. That is, the object detection device 10 of the second embodiment detects an object of an irregular shape by utilizing the difference in the reflectance of electromagnetic waves between background objects and the reflectance of electromagnetic waves from the object of an irregular shape, based on the intensity of the received signal shown in the reflected wave image. Then, an object of a fixed shape is detected by utilizing the shape of the object shown in the reflected wave image. The object detection device 10 of the second embodiment solves the problem of accurately detecting both objects of a fixed shape and objects of an irregular shape.

[0084] Furthermore, the object detection device 10 of the second embodiment generates a first likelihood map and a second likelihood map using a characteristic learning model as shown in Fig. 2. The learning model separately generates and outputs both the first likelihood map and the second likelihood map in response to input of a reflected wave image. A user can confirm the presence of an object with an irregular shape and an object with a fixed shape based on the output likelihood maps as shown in Figs. 7 and 8.

[0085] FIG. 3 shows the configuration of a comparative example. The comparative example uses two learning models: a first learning model that generates a first likelihood map in response to an input of a reflected wave image, and a second learning model that generates a second likelihood map in response to an input of a reflected wave image. That is, the comparative example uses two learning models to generate the first likelihood map and the second likelihood map. The configuration of this comparative example has problems such as increased computer resources. The object detection device 10 of the second embodiment can alleviate these problems.

[0086] Third Embodiment An object detection device 10 of a third embodiment identifies an area of ​​an object having an irregular shape in a reflected wave image based on a first likelihood map. The object detection device 10 also identifies an area of ​​an object having a fixed shape in a reflected wave image based on a second likelihood map. The object detection device 10 then outputs the identified area. This will be described in detail below.

[0087] 11 shows an example of a functional block diagram of the object detection device 10 according to the third embodiment. As shown in the figure, the object detection device 10 includes an image generation unit 11, a likelihood map generation unit 12, a first identification unit 13, a second identification unit 14, and an output unit 15.

[0088] 12 shows another example of a functional block diagram of the object detection device 10 according to the third embodiment. As shown in the figure, the object detection device 10 may further include a learning model generation unit 16 and a learning model storage unit 17.

[0089] The first identification unit 13 identifies a region of an irregularly shaped object in the reflected wave image based on the first likelihood map generated by the likelihood map generator 12. The first identification unit 13 can compare the likelihoods of a set of likelihood maps, such as those shown in Figures 7 and 8, for each pixel, and identify a group of pixels with the maximum likelihood corresponding to an irregularly shaped object as a region of the irregularly shaped object. As another example, the first identification unit 13 may identify, for example, a group of pixels whose likelihood corresponding to an irregularly shaped object shown in the first likelihood map is equal to or greater than a first reference value as a region of the irregularly shaped object.

[0090] The second identification unit 14 identifies a region of an object of a fixed shape in the reflected wave image based on the second likelihood map generated by the likelihood map generation unit 12. The second identification unit 14 can compare the likelihoods of a set of likelihood maps such as those shown in Fig. 7 and Fig. 8 with each other for each pixel, and identify a group of pixels with the maximum likelihood corresponding to an object of a fixed shape as the region of the object of a fixed shape.

[0091] As another example, the second identification unit 14 can identify, as the region of the fixed-shape object, a group of pixels whose likelihood corresponding to the fixed-shape object shown in the second likelihood map is equal to or greater than a second reference value. The first reference value and the second reference value may be the same value or different values.

[0092] As another example, the second identification unit 14 may identify, as the position of the fixed-shape object, a pixel where the likelihood corresponding to the fixed-shape object shown in the second likelihood map is a maximum value and where the value is equal to or greater than a third reference value.The second identification unit 14 may then identify a predetermined region based on the identified pixel as the region of the fixed-shape object.The "pixel where the likelihood is a maximum value" is a pixel with a higher likelihood than surrounding pixels.The "predetermined region based on the identified pixel" is, for example, a rectangular region centered on the identified pixel, but is not limited to this.

[0093] The output unit 15 generates and outputs the first image and the second image.

[0094] The "first image" indicates the area of ​​an object with an irregular shape in the reflected wave image identified by the first identification unit 13. The "second image" indicates the area of ​​an object with a fixed shape in the reflected wave image identified by the second identification unit 14.

[0095] Fig. 13 shows an example of the first image and the second image output by the output unit 15. The first image and the second image in Fig. 13 are generated based on the set of likelihood maps shown in Fig. 7.

[0096] The first image shown in Fig. 13 is created based on three likelihood maps corresponding to "blank (nothing)," "human body," and "object of irregular shape" from the set of likelihood maps shown in Fig. 7. The output unit 15 compares the likelihood of each of the three likelihood maps for each pixel and assigns to each pixel the type with the highest likelihood (blank (nothing), human body, object of irregular shape). Then, each pixel is displayed using a method (color, etc.) corresponding to the assigned type, thereby generating the first image shown in Fig. 13.

[0097] The second image shown in Fig. 13 is generated by superimposing information indicating the area of ​​the fixed-shaped object identified by the second identification unit 14 on the reflected wave image generated by the image generation unit 11. Note that in the example set of likelihood maps shown in Fig. 7, no fixed-shaped object was being held. Therefore, in the second image in Fig. 13, the reflected wave image is displayed as is, and information indicating the area of ​​the fixed-shaped object is not displayed.

[0098] As a modified example, if an object with a fixed shape is not detected in this way, information indicating this may be displayed on the second image. For example, any text information such as "Not Detected," "Undetected," or "OK" may be displayed, or any mark or figure may be displayed.

[0099] Fig. 14 shows another example of the first image and the second image output by the output unit 15. The first image and the second image in Fig. 14 are generated based on the set of likelihood maps shown in Fig. 8.

[0100] The first image shown in FIG. 14 is created based on three likelihood maps corresponding to "blank (nothing)," "human body," and "irregularly shaped object" from the set of likelihood maps shown in FIG. 8. The output unit 15 compares the likelihood of each of the three likelihood maps for each pixel and assigns the type with the highest likelihood (blank (nothing), human body, irregularly shaped object) to each pixel. Then, each pixel is displayed in a manner (color, etc.) corresponding to the assigned type, thereby generating the first image shown in FIG. 14. Note that in the example set of likelihood maps shown in FIG. 8, no irregularly shaped object was being held. Therefore, the area of ​​the irregularly shaped object is not shown in the first image of FIG. 14.

[0101] As a modified example, if an object of an irregular shape is not detected, information indicating this may be displayed on the first image. For example, any text information such as "Not detected," "Undetected," or "OK" may be displayed, or any mark or figure may be displayed.

[0102] The second image shown in Fig. 14 is generated by superimposing information indicating the area of ​​the fixed-shaped object identified by the second identification unit 14 on the reflected wave image generated by the image generation unit 11. In the example of the likelihood map set shown in Fig. 8, a firearm, which is an object with a fixed shape, was held. Therefore, in the second image of Fig. 14, information indicating that a firearm was detected and a frame indicating its position are superimposed on the reflected wave image.

[0103] Although not shown, as a modified example, the first image may be generated, similarly to the second image, by superimposing on the reflected wave image information indicating the region of the object having an irregular shape identified by the first identification unit 13. Furthermore, the second image may be generated by superimposing on a diagram showing the human body region, such as the first image, information indicating the region of the object having a fixed shape identified by the second identification unit 14.

[0104] Next, an example of the processing flow of the object detection device 10 will be described with reference to the flowchart of FIG.

[0105] First, the object detection device 10 generates a reflected wave image based on the received signal of the electromagnetic wave reflected by the object (S20).

[0106] Next, the object detection device 10 generates a first likelihood map and a second likelihood map based on the reflected wave image generated in S10 (S21). Specifically, the object detection device 10 generates a first likelihood map indicating the likelihood that each pixel in the reflected wave image corresponds to an object of an irregular shape based on the intensity of the received signal shown in the reflected wave image. The object detection device 10 also generates a second likelihood map indicating the likelihood that each pixel in the reflected wave image corresponds to an object of a predefined fixed shape based on the shape of the object shown in the reflected wave image.

[0107] Next, the object detection device 10 identifies an area of ​​an object having an irregular shape in the reflected wave image generated in S20 based on the first likelihood map generated in S21 (S22).Furthermore, the object detection device 10 identifies an area of ​​an object having a fixed shape in the reflected wave image generated in S20 based on the second likelihood map generated in S21 (S23).

[0108] Next, the object detection device 10 generates and outputs a first image showing the area of ​​the object of an irregular shape identified in S22 and a second image showing the area of ​​the object of a fixed shape identified in S23 (S24).

[0109] Other configurations of the object detection device 10 of the third embodiment are similar to those of the object detection device 10 of the first and second embodiments.

[0110] According to the object detection device 10 of the third embodiment, the same effects as those of the object detection devices 10 of the first and second embodiments are achieved.

[0111] Furthermore, the object detection device 10 of the third embodiment generates and outputs a first image indicating the position of an object of an irregular shape and a second image indicating the position of an object of a fixed shape. Based on the first image and the second image, the user can immediately determine whether an object of an irregular shape and an object of a fixed shape have been detected. Based on the first image and the second image, the user can also immediately determine the positions of the detected object of an irregular shape and an object of a fixed shape.

[0112] <Modification> In the second embodiment, the object detection device 10 includes the learning model generation unit 16, but the object detection device 10 does not have to include the learning model generation unit 16. In this case, an external device that is physically and / or logically separated from the object detection device 10 includes the learning model generation unit 16. Then, the learning model generated by the external device is stored in the learning model storage unit 17 by any means.

[0113] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations may be adopted. The configurations of the above-described embodiments may be combined with each other, or some of the configurations may be replaced with other configurations. Furthermore, various modifications may be made to the configurations of the above-described embodiments without departing from the spirit of the invention. Furthermore, the configurations and processes disclosed in the above-described embodiments and modified examples may be combined with each other.

[0114] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order. However, the order of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed to the extent that the content is not affected. Furthermore, the above embodiments can be combined to the extent that the content is not contradictory.

[0115] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes: 1. An object detection device comprising: an image generation means that generates a reflected wave image based on a received signal of electromagnetic waves reflected by an object; and a likelihood map generation means that generates a first likelihood map based on the intensity of the received signal shown in the reflected wave image, indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that occludes a background object and has a different reflectivity of electromagnetic waves from the background object, and that generates a second likelihood map based on the shape of the object shown in the reflected wave image, indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape. 2. The object detection device described in 1, further comprising: a first identification means that identifies a region of the object of the irregular shape based on the first likelihood map; and a second identification means that identifies a region of the object of the fixed shape based on the second likelihood map. 3. 3. The object detection device according to claim 2, further comprising an output means for generating and outputting a first image showing the region of the identified object of an irregular shape and a second image showing the region of the identified object of a fixed shape. 4. The object detection device according to any one of claims 1 to 3, wherein the likelihood map generation means generates the first likelihood map and the second likelihood map based on a learning model that outputs both the first likelihood map and the second likelihood map in response to input of the reflected wave image. 5. The object detection device according to claim 4, further comprising: a first learning data input means for inputting, as learning data relating to the irregular-shaped object, a first learning image and first ground truth data indicating the area of ​​the irregular-shaped object in the first learning image; a second learning data input means for inputting, as learning data relating to the fixed-shaped object, a second learning image and second ground truth data indicating the area of ​​the fixed-shaped object in the second learning image; and a learning means for generating the learning model based on the learning data relating to the irregular-shaped object and the learning data relating to the fixed-shaped object.6. The object detection device according to 5, wherein the first learning data input means detects, from the first learning image, a region where a predetermined feature amount obtained from image intensity satisfies a predetermined condition as the region of the irregular-shaped object, and uses the detection result as the first correct answer data. 7. The object detection device according to 5 or 6, wherein the first learning data input means detects a region of the irregular-shaped object in the first learning image based on the position of the irregular-shaped object identified in an image generated by a camera, and uses the detection result as the first correct answer data. 8. The object detection device according to any of 5 to 7, wherein the second learning data input means detects a region of the fixed-shaped object in the second learning image based on the position of the fixed-shaped object identified in an image generated by a camera, and uses the detection result as the second correct answer data. 9. The object detection device according to any of 1 to 8, wherein the irregular-shaped object includes a liquid or powdery substance. 10. The object detection device according to any of 1 to 9, wherein the background object is a human body. 11. The object detection device according to any one of 1 to 10, wherein the object of a fixed shape includes a firearm, a bladed weapon, or a weapon. 12. An object detection method, wherein one or more computers generate a reflected wave image based on received signals of electromagnetic waves reflected by an object, generate a first likelihood map based on the intensity of the received signals shown in the reflected wave image, indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that occludes a background object and has a reflectivity of electromagnetic waves different from that of the background object, and generate a second likelihood map based on the shape of the object shown in the reflected wave image, indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape.13. A program that causes a computer to function as: an image generating means that generates a reflected wave image based on received signals of electromagnetic waves reflected by an object; and a likelihood map generating means that generates a first likelihood map based on the intensity of the received signals shown in the reflected wave image, indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that shields a background object and has a different reflectivity of electromagnetic waves from the background object; and that generates a second likelihood map based on the shape of the object shown in the reflected wave image, indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape.

[0116] This application claims priority based on Japanese Patent Application No. 2022-197899, filed December 12, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0117] REFERENCE SIGNS LIST 10 Processing device 11 Image generation unit 12 Likelihood map generation unit 13 First identification unit 14 Second identification unit 15 Output unit 16 Learning model generation unit 17 Learning model storage unit 20 Learning device 21 First learning data input unit 22 Second learning data input unit 23 Learning unit 1A Processor 2A Memory 3A Input / output I / F 4A Peripheral circuit 5A Bus

Claims

1. an image generating means for generating a reflected wave image based on a received signal of the electromagnetic wave reflected by the object; a likelihood map generating means for generating a first likelihood map based on the intensity of the received signal shown in the reflected wave image, the first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that shields a background object and has a reflectivity of electromagnetic waves different from that of the background object, and for generating a second likelihood map based on the shape of the object shown in the reflected wave image, the second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape; An object detection device having:

2. a first identification means for identifying a region of the irregularly shaped object based on the first likelihood map; a second identification means for identifying a region of the fixed-shape object based on the second likelihood map; The object detection device according to claim 1 , further comprising:

3. 3. The object detection device according to claim 2, further comprising an output means for generating and outputting a first image showing the area of the identified object of an irregular shape and a second image showing the area of the identified object of a fixed shape.

4. The likelihood map generating means The object detection device described in claim 1, wherein the first likelihood map and the second likelihood map are generated based on a learning model that outputs both the first likelihood map and the second likelihood map in response to the input of the reflected wave image.

5. a first learning data input means for inputting, as learning data relating to the irregular-shaped object, a first learning image and first ground truth data indicating a region of the irregular-shaped object within the first learning image; a second learning data input means for inputting, as learning data relating to the fixed-shape object, a second learning image and second correct answer data indicating a region of the fixed-shape object within the second learning image; a learning means for generating the learning model based on learning data relating to the object of the irregular shape and learning data relating to the object of the fixed shape; The object detection device according to claim 4 , further comprising:

6. The first learning data input means The object detection device according to claim 5, wherein an area of the first learning image where a predetermined feature obtained from image intensity satisfies a predetermined condition is detected as an area of the irregularly shaped object, and the detection result is used as the first correct answer data.

7. The first learning data input means 6. The object detection device according to claim 5, wherein the area of the irregularly shaped object is detected in the first learning image based on the position of the irregularly shaped object identified in the image generated by the camera, and the detection result is used as the first correct answer data.

8. The second learning data input means includes: The object detection device according to claim 5, wherein the area of the fixed-shape object is detected in the second learning image based on the position of the fixed-shape object identified in the image generated by the camera, and the detection result is used as the second correct answer data.

9. One or more computers A reflected wave image is generated based on the received signal of the electromagnetic wave reflected by the object. a first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that shields a background object and has a reflectivity of electromagnetic waves different from that of the background object is generated based on the intensity of the received signal shown in the reflected wave image, and a second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape is generated based on the shape of the object shown in the reflected wave image; Object detection methods.

10. Computer, an image generating means for generating a reflected wave image based on a received signal of the electromagnetic wave reflected by the object; a likelihood map generating means for generating, based on the intensity of the received signal shown in the reflected wave image, a first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that shields a background object and has a reflectivity of electromagnetic waves different from that of the background object, and for generating, based on the shape of the object shown in the reflected wave image, a second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape; A program that functions as a