Object detection device, object detection method, and program

Abstract

The present invention provides an object detection device comprising: an image generation unit that generates a reflected wave image on the basis of received signals of electromagnetic waves reflected by objects; and a likelihood map generation unit that generates, on the basis of the intensity of the received signals indicated in the reflected wave image, a first likelihood map indicating the likelihood of each pixel in the reflected wave image corresponding to an irregularly shaped object occluding a background object and having differing electromagnetic wave reflectivity from the background object, and generates, on the basis of shapes of the objects shown in the reflected wave image, a second likelihood map indicating the likelihood of each pixel in the reflected wave image corresponding to a predefined fixed-shape object.

Description

[Technical Field]

[0001] The present invention relates to an object detection device, an object detection method, and a program. [Background technology]

[0002] Techniques related to the present invention are disclosed in Patent Documents 1 to 4. The techniques disclosed in Patent Documents 1 to 4 relate to inspection of belongings using electromagnetic waves.

[0003] The technology disclosed in Patent Document 1 uses data on the intensity and depth along the surface of the object to check whether a target person is carrying artificial objects suspected of being contraband or a security threat (such as explosives).

[0004] Patent Document 2 discloses that it may be possible to visualize an object in an X-ray image of the human body due to differences in X-ray reflectivity between the object and human tissue.

[0005] The technology disclosed in Patent Document 3 identifies the type of object contained in a detection target from the difference between the radiant energy signal emitted by the detection target and the radiant energy signal of the detection target surrounding the non-detection target. The technology disclosed in Patent Document 3 can inspect a wide range of substances, from powders to solids such as metals.

[0006] Patent Document 4 discloses that the reflectance of electromagnetic waves reflected by the human body differs from that of electromagnetic waves reflected by an object concealed in clothing, and that the concealed object can be detected based on this difference. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Special Publication No. 2007-517275 [Patent Document 2] Special Publication No. 2010-530977 [Patent Document 3] Japanese Patent Application Laid-Open No. 2017-40593 [Patent Document 4] Patent Publication No. 2021-32778 Summary of the Invention [Problem to be solved by the invention]

[0008] In a personal possession search, there are cases where both fixed-shaped objects and irregular-shaped objects are to be detected. Fixed-shaped objects include, for example, guns and blades. Irregular-shaped objects include, for example, liquids and powders. Therefore, there is a demand for technology that can accurately detect both fixed-shaped objects and irregular-shaped objects.

[0009] The techniques described in Patent Documents 1 to 4 can detect a wide range of objects. However, these techniques use the same method to detect both objects with fixed shapes and objects with irregular shapes. Unless a method suited to each object is used, the detection accuracy will be poor.

[0010] In view of the above-mentioned problems, one example of the object of the present invention is to provide an object detection device, an object detection method, and a program that solve the problem of accurately detecting both objects with fixed shapes and objects with irregular shapes. [Means for solving the problem]

[0011] According to one aspect of the present invention, an image generating means for generating a reflected wave image based on a received signal of the electromagnetic wave reflected by the object; a likelihood map generating means for generating a first likelihood map based on the intensity of the received signal shown in the reflected wave image, the first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that shields a background object and has a reflectivity of electromagnetic waves different from that of the background object, and for generating a second likelihood map based on the shape of the object shown in the reflected wave image, the second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape; An object detection device is provided, comprising:

[0012] According to one aspect of the present invention, One or more computers A reflected wave image is generated based on the received signal of the electromagnetic wave reflected by the object. a first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that shields a background object and has a reflectivity of electromagnetic waves different from that of the background object is generated based on the intensity of the received signal shown in the reflected wave image, and a second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape is generated based on the shape of the object shown in the reflected wave image; A method for detecting an object is provided.

[0013] According to one aspect of the present invention, Computer, an image generating means for generating a reflected wave image based on a received signal of the electromagnetic wave reflected by the object; a likelihood map generating means for generating, based on the intensity of the received signal shown in the reflected wave image, a first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that shields a background object and has a reflectivity of electromagnetic waves different from that of the background object, and for generating, based on the shape of the object shown in the reflected wave image, a second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape; A program is provided to function as a [Effects of the Invention]

[0014] According to one aspect of the present invention, an object detection device, an object detection method, and a program are realized that solve the problem of accurately detecting both objects with fixed shapes and objects with irregular shapes. [Brief explanation of the drawings]

[0015] The above and other objects, features and advantages are described below. Suitable This will become more apparent from the following embodiments and the accompanying drawings.

[0016] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional block diagram of an object detection device. [Figure 2] FIG. 2 is a diagram for explaining the characteristics of a learning model used by the object detection device. [Figure 3] FIG. 10 is a diagram for explaining a comparative example. [Figure 4] FIG. 1 is a diagram illustrating an example of a hardware configuration of an object detection device. [Figure 5] FIG. 2 is a diagram illustrating an example of a functional block diagram of an object detection device. [Figure 6] FIG. 2 is a diagram illustrating an example of a reflected wave image generated by the object detection device. [Figure 7] FIG. 10 is a diagram illustrating an example of a set of likelihood maps generated by the object detection device. [Figure 8] FIG. 10 is a diagram illustrating another example of a set of likelihood maps generated by the object detection device. [Figure 9] FIG. 2 is a diagram illustrating an example of a functional block diagram of a learning device. [Figure 10] 10 is a flowchart illustrating an example of a processing flow of the object detection device. [Figure 11] FIG. 10 is a diagram illustrating another example of a functional block diagram of the object detection device. [Figure 12] FIG. 10 is a diagram illustrating another example of a functional block diagram of the object detection device. [Figure 13] 3A to 3C are diagrams illustrating an example of a first image and a second image generated by the object detection device. [Figure 14] 5A and 5B are diagrams showing other examples of the first image and the second image generated by the object detection device. [Figure 15] 10 is a flowchart illustrating an example of a processing flow of the object detection device. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0018] First Embodiment 1 is a functional block diagram showing an overview of an object detection device 10 according to the first embodiment. The object detection device 10 includes an image generation unit 11 and a likelihood map generation unit 12.

[0019] The image generating unit 11 generates a reflected wave image based on the received signal of the electromagnetic wave reflected by the object.

[0020] The likelihood map generator 12 generates a first likelihood map and a second likelihood map.

[0021] The first likelihood map indicates the likelihood that each pixel of the reflected wave image corresponds to an irregularly shaped object that shields background objects and has a different reflectivity of electromagnetic waves than the background objects (hereinafter, sometimes simply referred to as an "irregularly shaped object"). That is, the first likelihood map indicates the likelihood that each pixel of the reflected wave image indicates the intensity of the reflected wave reflected by the irregularly shaped object. The likelihood map generator 12 generates the first likelihood map based on the intensity of the received signal indicated in the reflected wave image, using the difference between the reflectivity of the electromagnetic waves of the background objects and the reflectivity of the electromagnetic waves of the irregularly shaped object.

[0022] The second likelihood map indicates the likelihood that each pixel in the reflected wave image corresponds to an object of a predefined fixed shape. That is, the second likelihood map indicates the likelihood that each pixel in the reflected wave image indicates the intensity of a reflected wave reflected by an object of a fixed shape. The likelihood map generator 12 generates the second likelihood map using the shape of the object shown in the reflected wave image.

[0023] According to the object detection device 10 of the first embodiment, an object of an irregular shape is detected based on the intensity of the received signal shown in the reflected wave image, utilizing the difference in the reflectance of the electromagnetic wave between the background object and the object of an irregular shape. Furthermore, an object of a fixed shape is detected using the shape of the object shown in the reflected wave image. According to the object detection device 10 of the first embodiment, the problem of accurately detecting both an object of a fixed shape and an object of an irregular shape is solved.

[0024] <Second embodiment> "overview" The object detection device 10 of the second embodiment is a specific implementation of the configuration of the object detection device 10 of the first embodiment. The object detection device 10 of the second embodiment has the same configuration as the object detection device 10 of the first embodiment, and achieves the same effects. In addition, the object detection device 10 of the second embodiment has the following features.

[0025] As described in the first embodiment, the object detection device 10 generates a first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape, and a second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a fixed shape.

[0026] The object detection device 10 then generates both the first likelihood map and the second likelihood map using a previously generated learning model. As shown in Fig. 2, the learning model receives an input of a reflected wave image. The learning model then separately generates and outputs both the first likelihood map and the second likelihood map in response to the input of the reflected wave image.

[0027] FIG. 3 shows the configuration of a comparative example. The comparative example uses two models: a first learning model that generates a first likelihood map in response to an input of a reflected wave image, and a second learning model that generates a second likelihood map in response to an input of a reflected wave image. That is, the comparative example uses two learning models to generate the first likelihood map and the second likelihood map. The configuration of this comparative example has problems such as increased computer resources. The object detection device 10 of the second embodiment can alleviate these problems.

[0028] The configuration of the object detection device 10 according to the second embodiment will be described in detail below.

[0029] "Hardware Configuration" Next, an example of the hardware configuration of the object detection device 10 will be described. Each functional unit of the object detection device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various variations in the realization method and device. The software includes programs that are pre-loaded when the device is shipped, and programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet.

[0030] FIG. 4 is a block diagram illustrating an example of the hardware configuration of the object detection device 10. As shown in FIG. 4, the object detection device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The object detection device 10 does not necessarily have to have the peripheral circuit 4A. Note that the object detection device 10 may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices can have the above hardware configuration.

[0031] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to mutually transmit and receive data. The processor 1A is, for example, a processing unit such as a CPU or a graphics processing unit (GPU). The memory 2A is, for example, a random access memory (RAM) or a read-only memory (ROM). The input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. The input / output interface 3A also includes an interface for connecting to a communication network such as the Internet. Examples of input devices include a keyboard, mouse, microphone, physical buttons, and touch panel. Examples of output devices include a display, speaker, printer, and mailer. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0032] "Function Configuration" Next, the functional configuration of the object detection device 10 according to the second embodiment will be described in detail. An example of a functional block diagram of the object detection device 10 is shown in FIG. 5. As shown in the figure, the object detection device 10 includes an image generation unit 11, a likelihood map generation unit 12, an output unit 15, a learning model generation unit 16, and a learning model storage unit 17.

[0033] The image generating unit 11 generates a reflected wave image based on the received signal of the electromagnetic wave reflected by the object.

[0034] In the "reflected wave image," each pixel indicates the intensity (intensity of the received signal) of the electromagnetic wave (reflected wave) reflected by an object. FIG. 6 shows an example of a reflected wave image. The illustrated reflected wave image shows a human body. The generation of a reflected wave image based on the received electromagnetic wave signal can be achieved using any known technology.

[0035] The "object" is a human body, possessions carried by the human body, etc. The object detection device 10 is used, for example, in a possession search. In this case, the object is the human body of the person who is the subject of the possession search and the possessions carried by that person.

[0036] The "electromagnetic wave reception signal" is generated by an electromagnetic wave transmission / reception device. The object detection device 10 may be equipped with an electromagnetic wave transmission / reception device. Alternatively, the electromagnetic wave transmission / reception device may be a separate device that is physically and / or logically separated from the object detection device 10. The electromagnetic wave reception signal generated by the electromagnetic wave transmission / reception device may be input to the object detection device 10 by any means.

[0037] The electromagnetic wave transmitting and receiving device controls a transmitting antenna that radiates electromagnetic waves such as millimeter waves and a receiving antenna that receives reflected waves of the electromagnetic waves radiated from the transmitting antenna. The electromagnetic wave transmitting and receiving device controls, for example, the radiation of the electromagnetic waves from the transmitting antenna, specifically the radiation timing, etc.

[0038] The electromagnetic waves emitted by the transmitting antenna can be, for example, continuous wave (CW), frequency modulated continuous wave (FMCW), or stepped frequency continuous wave (SFCW). The receiving antenna measures the complex amplitude of the received wave (a complex number representing the amplitude and phase shift from the transmitted wave) for each frequency, and the measurement results are used as a radar signal. The radar signal can be expressed as S(n,m,f), where n is the transmitting antenna number, m is the receiving antenna number, and f is the frequency.

[0039] The antenna configuration is not particularly limited and any configuration can be used, for example, an antenna panel in which multiple transmitting antennas and multiple receiving antennas are arranged may be used.

[0040] The multiple antennas are installed in positions and orientations that irradiate electromagnetic waves into a target space and receive waves reflected by objects located within the target space.

[0041] The target space is a space where a predetermined inspection, such as a possessions inspection, is carried out. The above-mentioned antenna irradiates electromagnetic waves onto the target space including the target object and receives the reflected waves. Various inspections are then carried out based on the radar signals of the reflected waves. The target space is provided, for example, in a passageway through which the target object passes. Then, while the target object is moving within the target space, electromagnetic waves are irradiated and the reflected waves are received. In this way, in one example of the second embodiment, a walk-through type inspection is realized. Alternatively, the target space may be a space where the target object will stop for a certain period of time for inspection.

[0042] Returning to FIG. 5, the likelihood map generator 12 generates a first likelihood map and a second likelihood map.

[0043] The "first likelihood map" indicates the likelihood that each pixel of the reflected wave image corresponds to an irregularly shaped object that occludes a background object and has a different reflectivity of electromagnetic waves than the background object. In other words, the first likelihood map indicates the likelihood that each pixel of the reflected wave image indicates the intensity of the reflected wave reflected by the irregularly shaped object.

[0044] The likelihood map generator 12 generates a first likelihood map based on the intensity of the received signal shown in the reflected wave image, utilizing the difference between the reflectance of the electromagnetic wave of the background object and the reflectance of the electromagnetic wave of the irregularly shaped object. The irregularly shaped object to be detected overlaps with the background object in the reflected wave image, occluding it. The reflectance of the electromagnetic wave of the irregularly shaped object differs from the reflectance of the electromagnetic wave of the background object. Due to this characteristic, the intensity of the received signal of a portion of the background object (a portion occluded by the irregularly shaped object) in the reflected wave image is lower than that of other portions (a portion not occluded by the irregularly shaped object). Based on this phenomenon, the likelihood that each pixel in the reflected wave image corresponds to an irregularly shaped object can be calculated.

[0045] The "background object" is, for example, a human body. The "object of an irregular shape" includes, for example, a liquid or powder. The liquid is a liquid that is prohibited from being possessed or brought into a facility. The powder is a powder-like substance that is prohibited from being possessed or brought into a facility, such as narcotics.

[0046] The "second likelihood map" indicates the likelihood that each pixel in the reflected wave image corresponds to an object of a predefined fixed shape. In other words, the second likelihood map indicates the likelihood that each pixel in the reflected wave image indicates the intensity of the reflected wave reflected by an object of a predefined fixed shape.

[0047] The likelihood map generator 12 generates a second likelihood map based on the shape of the object shown in the reflected wave image. A fixed-shape object has a predetermined shape. Therefore, based on the shape of the object shown in the reflected wave image and the shape of the predefined fixed-shape object, it is possible to calculate the likelihood that each pixel in the reflected wave image corresponds to the predefined fixed-shape object.

[0048] "Fixed-shape objects" include firearms, knives, and other weapons. A predetermined type of object is defined in advance as a fixed-shape object. There may be one or more fixed-shape objects defined in advance.

[0049] The likelihood map generation unit 12 generates both a first likelihood map and a second likelihood map separately in response to the input of a reflected wave image, and generates the first likelihood map and the second likelihood map based on the output learning model (see Figure 2).

[0050] When a reflected wave image is input to the learning model, multiple likelihood maps such as those shown in FIG. 7 are output. In the example of FIG. 7, the learning model generates and outputs five likelihood maps in response to the input reflected wave image. Specifically, likelihood maps corresponding to "firearms," ​​"swords," "blank (nothing)," "human body," and "objects of irregular shape" are output. In FIG. 7, the likelihood of the corresponding object type is displayed in shades of black and white, with areas with high likelihood being white and areas with low likelihood being black. In the example shown in FIG. 7, two objects, "firearms" and "swords," are defined in advance as objects with fixed shapes. Then, in response to the input reflected wave image, the learning model generates and outputs likelihood maps (second likelihood maps) corresponding to each of the two objects with fixed shapes.

[0051] The likelihood map corresponding to "gun" indicates the likelihood that each pixel of the reflected wave image indicates the intensity of the reflected wave reflected by the gun. The likelihood map corresponding to "sword" indicates the likelihood that each pixel of the reflected wave image indicates the intensity of the reflected wave reflected by the sword. In the example of Figure 7, these fixed-shaped objects were not being held. Therefore, in both likelihood maps, the likelihood that each pixel of the reflected wave image indicates the intensity of the reflected wave reflected by the gun and the sword is low.

[0052] The likelihood map corresponding to "blank (nothing)" indicates the likelihood that there was nothing at the position of each pixel in the reflected wave image.

[0053] The likelihood map corresponding to the "human body" indicates the likelihood that each pixel of the reflected wave image indicates the intensity of the reflected wave reflected by the human body. The likelihood may be calculated based on, for example, the shape of the human body.

[0054] The likelihood map (first likelihood map) corresponding to the "irregularly shaped object" indicates the likelihood that each pixel in the reflected wave image indicates the intensity of the reflected wave reflected by the irregularly shaped object. In the example of Figure 7, an irregularly shaped object was present. Therefore, some pixels in the reflected wave image have a high likelihood of indicating the intensity of the reflected wave reflected by the irregularly shaped object.

[0055] FIG. 8 shows another example of a likelihood map output from the learning model. In FIG. 8, as in FIG. 7, the likelihood of the corresponding object type is displayed in shades of black and white, with high likelihood areas being white and low likelihood areas being black. In this example, a firearm, which is an object of a fixed shape, was being carried. Therefore, in the likelihood map corresponding to the firearm (second likelihood map), some pixels in the reflected wave image have a high likelihood indicating the intensity of the reflected wave reflected by the firearm. Note that in the example of FIG. 8, neither a sword nor an object of an irregular shape was being carried. Therefore, in both likelihood maps, the likelihood of each pixel in the reflected wave image indicating the intensity of the reflected wave reflected by the sword and the object of an irregular shape is low.

[0056] Returning to FIG. 5, the learning model generation unit 16 generates the above-described learning model and stores it in the learning model storage unit 17. The learning model generation unit 16 has the configuration of a learning device 20 described below. As shown in FIG. 9, the learning device 20 has a first learning data input unit 21, a second learning data input unit 22, and a learning unit 23. An example of the hardware configuration of the learning device 20 is shown in FIG. 4.

[0057] The first learning data input unit 21 inputs a first learning image and first correct answer data to the learning device 20 as learning data relating to an object of an irregular shape.

[0058] In the "first learning image," each pixel represents the intensity of the reflected electromagnetic wave (the intensity of the received signal). The first learning image represents the intensity of the electromagnetic wave reflected by an object of an irregular shape and the intensity of the electromagnetic wave reflected by a human body holding the object of an irregular shape. When generating the first learning image, electromagnetic waves are irradiated toward a human body holding an object of an irregular shape, and the reflected waves are received. Then, the first learning image is generated based on the intensity of the received signal (the intensity of the reflected electromagnetic wave).

[0059] The "first correct answer data" indicates the region of the irregularly shaped object in the first training image. The first correct answer data may be generated manually by the user. That is, the user may visually recognize the first training image, identify the region of the irregularly shaped object in the first training image, and input the identified region to the training device 20.

[0060] Alternatively, the first training data input unit 21 may generate the first supervised data from the first training images without the manual operation by the user as described above. An example of the process in which the first training data input unit 21 generates the first supervised data from the first training images will be described below.

[0061] - Example 1 of a process for generating first ground truth data from a first learning image - For example, the first learning data input unit 21 detects, from the first learning image, an area where a predetermined feature amount obtained from image intensity (strength of the received signal) satisfies a predetermined condition as an area of ​​an object of an irregular shape. Then, the first learning data input unit 21 uses the detection result (data indicating the detected area) as first ground truth data.

[0062] The "predetermined feature amount" is, for example, the dielectric constant. The first training data input unit 21 processes the first training image to generate a dielectric constant image in which the influence of position, etc., has been corrected. The dielectric constant image indicates the dielectric constant corresponding to each pixel. The first training data input unit 21 then detects regions where the dielectric constant is equal to or less than a threshold as regions of irregularly shaped objects. In this method, regions of irregularly shaped objects are detected by utilizing the fact that the dielectric constant of irregularly shaped objects is equal to or less than a threshold.

[0063] Alternatively, the predetermined feature may be the intensity of the received signal itself shown in the reflected wave image. In this case, the first learning data input unit 21 detects, as an area of ​​an object of an irregular shape, an area in which the intensity of the received signal is within a predetermined numerical range (a value calculated based on the reflectance of the object of an irregular shape, etc.).

[0064] - Example 2 of a process for generating first correct answer data from a first learning image - In this example, the electromagnetic wave transmitting and receiving device is used to irradiate an object with electromagnetic waves, and the reflected waves are received. At the same time, a camera (e.g., a camera equipped with a sensor that detects visible light) captures an image of the object (hereinafter, "camera image"), and the electromagnetic wave transmitting and receiving device and the camera are installed in approximately the same position and capture the object from approximately the same direction.

[0065] The first training data input unit 21 identifies the position of an object having an irregular shape in a camera image generated by a camera. Then, the first training data input unit 21 identifies a position in the first training image that corresponds to the position of the object having an irregular shape identified in the camera image, and detects the identified position in the first training image as the region of the object having an irregular shape.

[0066] One method for identifying the position of an irregularly shaped object in a camera image is to use a marker. That is, when photographing a human body holding an irregularly shaped object, a marker that clearly indicates the position of the irregularly shaped object is captured in the camera image. The marker may be, for example, a sticker with a predetermined mark. In this case, the sticker is attached to the position of the irregularly shaped object. The detection of the marker in the camera image is achieved using conventional image analysis technology.

[0067] Next, a process for detecting an area of ​​an object of an irregular shape in the first learning image based on the position of the object of an irregular shape identified in the camera image will be described.

[0068] As described above, the camera image and the first learning image are images of a human body holding an irregularly shaped object captured from approximately the same position and approximately the same direction. Therefore, the relative positional relationship between the human body and the irregularly shaped object shown in the camera image is equivalent to the relative positional relationship between the human body and the irregularly shaped object in the first learning image. Based on this relationship, the first learning data input unit 21 identifies a "position in the first learning image" that corresponds to the "position of the irregularly shaped object identified in the camera image." The first learning data input unit 21 then determines a predetermined region based on the identified position in the first learning image as the region of the irregularly shaped object in the first learning image. An example of a predetermined region based on the identified position in the first learning image is a rectangular region centered on the identified position, but is not limited to this.

[0069] The second learning data input unit 22 inputs the second learning image and the second correct answer data to the learning device 20 as learning data relating to an object of a fixed shape.

[0070] In the "second training image," each pixel represents the intensity of the reflected electromagnetic wave (the intensity of the received signal). The second training image represents the intensity of the electromagnetic wave reflected by an object of a fixed shape and the intensity of the electromagnetic wave reflected by a human body holding the object of the fixed shape. When generating the second training image, electromagnetic waves are irradiated toward a human body holding an object of a fixed shape, and the reflected waves are received. Then, the second training image is generated based on the intensity of the received signal (the intensity of the reflected electromagnetic wave).

[0071] The "second correct answer data" indicates the area of ​​the fixed-shape object in the second training image. The second correct answer data may be generated manually by the user. That is, the user may visually recognize the second training image, identify the area of ​​the fixed-shape object in the second training image, and input the identified area to the training device 20.

[0072] Alternatively, the second training data input unit 22 may generate the second correct answer data from the second training images without the manual operation by the user as described above.

[0073] For example, the electromagnetic wave transmitting and receiving device described above is used to irradiate an object with electromagnetic waves, and the reflected waves are received, and the object is photographed with a camera (e.g., a camera equipped with a sensor that detects visible light) to generate a camera image. The electromagnetic wave transmitting and receiving device and the camera are installed in approximately the same position and capture the object from approximately the same direction.

[0074] The second training data input unit 22 identifies the position of an object of a fixed shape in a camera image generated by the camera. Then, the second training data input unit 22 identifies a position in the second training image that corresponds to the position of the object of a fixed shape identified in the camera image, and detects the identified position in the second training image as the region of the object of a fixed shape.

[0075] One method for identifying the position of a fixed-shape object in a camera image is to use a marker. That is, when a human body holding a fixed-shape object is photographed, a marker that clearly indicates the position of the fixed-shape object is captured in the camera image. The marker may be, for example, a sticker with a predetermined mark. In this case, the sticker is attached to the position of the fixed-shape object. The detection of the marker in the camera image is achieved using conventional image analysis technology.

[0076] Next, a process for detecting the region of the fixed-shape object in the second learning image based on the position of the fixed-shape object identified in the camera image will be described.

[0077] As described above, the camera image and the second training image are images of a human body holding a fixed-shape object captured from approximately the same position and approximately the same direction. Therefore, the relative positional relationship between the human body and the fixed-shape object shown in the camera image is equivalent to the relative positional relationship between the human body and the fixed-shape object in the second training image. Based on this relationship, the second training data input unit 22 identifies a "position in the second training image" that corresponds to the "position of the fixed-shape object identified in the camera image." The second training data input unit 22 then determines a predetermined region based on the identified position in the second training image as the region of the fixed-shape object in the second training image. An example of a predetermined region based on the identified position in the second training image is, but is not limited to, a rectangular region centered on the identified position.

[0078] The learning unit 23 generates the above-mentioned learning model by learning based on the learning data relating to objects of irregular shapes input by the first learning data input unit 21 and the learning data relating to objects of fixed shapes input by the second learning data input unit 22. There are no particular limitations on the learning method.

[0079] Next, an example of the processing flow of the object detection device 10 will be described with reference to the flowchart of FIG.

[0080] First, the object detection device 10 generates a reflected wave image based on the received signal of the electromagnetic wave reflected by the object (S10).

[0081] Next, the object detection device 10 generates a first likelihood map and a second likelihood map based on the reflected wave image generated in S10 (S11). Specifically, the object detection device 10 generates a first likelihood map indicating the likelihood that each pixel in the reflected wave image corresponds to an object of an irregular shape based on the intensity of the received signal shown in the reflected wave image. The object detection device 10 also generates a second likelihood map indicating the likelihood that each pixel in the reflected wave image corresponds to an object of a predefined fixed shape based on the shape of the object shown in the reflected wave image.

[0082] Then, the object detection device 10 outputs the generated first likelihood map and second likelihood map (S12). For example, the object detection device 10 outputs a set of likelihood maps as shown in Fig. 7 or 8. The object detection device 10 outputs the maps via any output device such as a display, a projection device, a printer, or the like.

[0083] "Action and effect" The object detection device 10 of the second embodiment achieves the same effects as the object detection device 10 of the first embodiment. That is, the object detection device 10 of the second embodiment detects an object of an irregular shape by utilizing the difference in the reflectance of electromagnetic waves between background objects and the reflectance of electromagnetic waves from an object of an irregular shape, based on the intensity of the received signal shown in the reflected wave image. Then, an object of a fixed shape is detected by utilizing the shape of the object shown in the reflected wave image. The object detection device 10 of the second embodiment solves the problem of accurately detecting both objects of a fixed shape and objects of an irregular shape.

[0084] Furthermore, the object detection device 10 of the second embodiment generates a first likelihood map and a second likelihood map using a characteristic learning model as shown in Fig. 2. The learning model separately generates and outputs both the first likelihood map and the second likelihood map in response to input of a reflected wave image. A user can confirm the presence of an object with an irregular shape and an object with a fixed shape based on the output likelihood maps as shown in Figs. 7 and 8.

[0085] FIG. 3 shows the configuration of a comparative example. The comparative example uses two models: a first learning model that generates a first likelihood map in response to an input of a reflected wave image, and a second learning model that generates a second likelihood map in response to an input of a reflected wave image. That is, the comparative example uses two learning models to generate the first likelihood map and the second likelihood map. The configuration of this comparative example has problems such as increased computer resources. The object detection device 10 of the second embodiment can alleviate these problems.

[0086] <Third embodiment> The object detection device 10 of the third embodiment identifies an area of ​​an object with an irregular shape in the reflected wave image based on a first likelihood map. The object detection device 10 also identifies an area of ​​an object with a fixed shape in the reflected wave image based on a second likelihood map. The object detection device 10 then outputs the identified results. This will be described in detail below.

[0087] 11 shows an example of a functional block diagram of an object detection device 10 according to the third embodiment. As shown in the figure, the object detection device 10 includes an image generation unit 11, a likelihood map generation unit 12, a first identification unit 13, a second identification unit 14, and an output unit 15.

[0088] 12 shows another example of a functional block diagram of the object detection device 10 according to the third embodiment. As shown in the figure, the object detection device 10 may further include a learning model generation unit 16 and a learning model storage unit 17.

[0089] The first identification unit 13 identifies a region of an irregularly shaped object in the reflected wave image based on the first likelihood map generated by the likelihood map generation unit 12. The first identification unit 13 can compare the likelihoods of a set of likelihood maps such as those shown in Figures 7 and 8 for each pixel, and identify a group of pixels with the maximum likelihood corresponding to an irregularly shaped object as the region of the irregularly shaped object. As another example, the first identification unit 13 may identify a group of pixels with a likelihood corresponding to an irregularly shaped object shown in the first likelihood map that is equal to or greater than a first reference value as the region of the irregularly shaped object.

[0090] The second identification unit 14 identifies a region of an object of a fixed shape in the reflected wave image based on the second likelihood map generated by the likelihood map generation unit 12. The second identification unit 14 can compare the likelihoods of a set of likelihood maps such as those shown in Fig. 7 and Fig. 8 for each pixel, and identify a group of pixels with the maximum likelihood corresponding to an object of a fixed shape as the region of the object of a fixed shape.

[0091] As another example, the second identification unit 14 can identify a group of pixels whose likelihood corresponding to a fixed-shape object shown in the second likelihood map is equal to or greater than a second reference value as the region of the fixed-shape object. The first reference value and the second reference value may be the same value or different values.

[0092] As another example, the second identification unit 14 may identify a pixel where the likelihood corresponding to the fixed-shape object shown in the second likelihood map is a maximum value and where the value is equal to or greater than a third reference value as the position of the fixed-shape object.The second identification unit 14 may then identify a predetermined area based on the identified pixel as the area of ​​the fixed-shape object.The "pixel where the likelihood is a maximum value" is a pixel with a higher likelihood than surrounding pixels.The "predetermined area based on the identified pixel" is, for example, a rectangular area centered on the identified pixel, but is not limited to this.

[0093] The output unit 15 generates and outputs the first image and the second image.

[0094] The "first image" indicates the area of ​​an object of an irregular shape in the reflected wave image identified by the first identification unit 13. The "second image" indicates the area of ​​an object of a fixed shape in the reflected wave image identified by the second identification unit 14.

[0095] 13 shows an example of the first image and the second image output by the output unit 15. The first image and the second image in FIG. 13 are generated based on the set of likelihood maps shown in FIG.

[0096] The first image shown in Fig. 13 is created based on three likelihood maps corresponding to "blank (nothing)," "human body," and "object of irregular shape" from the set of likelihood maps shown in Fig. 7. The output unit 15 compares the likelihood of each of the three likelihood maps for each pixel, and assigns to each pixel the type with the highest likelihood (blank (nothing), human body, object of irregular shape). Then, each pixel is displayed in a manner (color, etc.) corresponding to the assigned type, thereby generating the first image shown in Fig. 13.

[0097] The second image shown in Fig. 13 is generated by superimposing information indicating the area of ​​the fixed-shaped object identified by the second identification unit 14 on the reflected wave image generated by the image generation unit 11. Note that in the example of the likelihood map set shown in Fig. 7, no fixed-shaped object was being held. Therefore, in the second image in Fig. 13, the reflected wave image is displayed as is, and information indicating the area of ​​the fixed-shaped object is not displayed.

[0098] As a modified example, if an object with a fixed shape is not detected in this way, information indicating this may be displayed on the second image. For example, any text information such as "Not Detected," "Undetected," or "OK" may be displayed, or any mark or figure may be displayed.

[0099] 14 shows another example of the first image and the second image output by the output unit 15. The first image and the second image in FIG. 14 are generated based on the set of likelihood maps shown in FIG.

[0100] The first image shown in FIG. 14 is created based on three likelihood maps corresponding to "blank (nothing)," "human body," and "irregularly shaped object" from the set of likelihood maps shown in FIG. 8. The output unit 15 compares the likelihood of each of the three likelihood maps for each pixel and assigns to each pixel the type with the highest likelihood (blank (nothing), human body, irregularly shaped object). Then, each pixel is displayed in a manner (color, etc.) corresponding to the assigned type, thereby generating the first image shown in FIG. 14. Note that in the example set of likelihood maps shown in FIG. 8, no irregularly shaped object was being held. Therefore, the area of ​​the irregularly shaped object is not shown in the first image of FIG. 14.

[0101] As a modified example, if an object of an irregular shape is not detected, information indicating this may be displayed on the first image. For example, any text information such as "Not Detected," "Undetected," or "OK" may be displayed, or any mark or figure may be displayed.

[0102] The second image shown in Fig. 14 is generated by superimposing information indicating the area of ​​the fixed-shaped object identified by the second identification unit 14 on the reflected wave image generated by the image generation unit 11. In the example of the likelihood map set shown in Fig. 8, a firearm, which is an object with a fixed shape, was held. Therefore, in the second image of Fig. 14, information indicating that a firearm was detected and a frame indicating its position are superimposed on the reflected wave image.

[0103] Although not shown, as a modified example, the first image may be generated, similarly to the second image, by superimposing on the reflected wave image information indicating the region of an object of an irregular shape identified by the first identification unit 13. Furthermore, the second image may be generated by superimposing on a diagram showing a human body region such as the first image information indicating the region of an object of a fixed shape identified by the second identification unit 14.

[0104] Next, an example of the processing flow of the object detection device 10 will be described with reference to the flowchart of FIG.

[0105] First, the object detection device 10 generates a reflected wave image based on the received signal of the electromagnetic wave reflected by the object (S20).

[0106] Next, the object detection device 10 2 Based on the reflected wave image generated in step S20, the object detection device 10 generates a first likelihood map and a second likelihood map (S21). Specifically, the object detection device 10 generates a first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape based on the intensity of the received signal shown in the reflected wave image. The object detection device 10 also generates a second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape based on the shape of the object shown in the reflected wave image.

[0107] Next, the object detection device 10 identifies an area of ​​an object having an irregular shape in the reflected wave image generated in S20 based on the first likelihood map generated in S21 (S22). Also, the object detection device 10 identifies an area of ​​an object having a fixed shape in the reflected wave image generated in S20 based on the second likelihood map generated in S21 (S23).

[0108] Next, the object detection device 10 generates and outputs a first image showing the region of the object of an irregular shape identified in S22 and a second image showing the region of the object of a fixed shape identified in S23 (S24).

[0109] Other configurations of the object detection device 10 of the third embodiment are similar to those of the object detection device 10 of the first and second embodiments.

[0110] According to the object detection device 10 of the third embodiment, the same effects as those of the object detection devices 10 of the first and second embodiments are achieved.

[0111] Furthermore, the object detection device 10 of the third embodiment generates and outputs a first image indicating the position of an object of an irregular shape and a second image indicating the position of an object of a fixed shape. Based on the first and second images, the user can immediately determine whether an object of an irregular shape and an object of a fixed shape have been detected. Based on the first and second images, the user can also immediately determine the positions of the detected object of an irregular shape and an object of a fixed shape.

[0112] <Modification> In the second embodiment, the object detection device 10 is provided with the learning model generation unit 16, but the object detection device 10 does not have to be provided with the learning model generation unit 16. In this case, an external device that is physically and / or logically separated from the object detection device 10 includes the learning model generation unit 16. Then, the learning model generated by the external device is stored in the learning model storage unit 17 by any means.

[0113] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations may be adopted. The configurations of the above-described embodiments may be combined with each other, or some of the configurations may be replaced with other configurations. Furthermore, various modifications may be made to the configurations of the above-described embodiments without departing from the spirit of the invention. Furthermore, the configurations and processes disclosed in the above-described embodiments and modified examples may be combined with each other.

[0114] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order. However, the order of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not cause any problems in terms of content. Furthermore, the above embodiments can be combined as long as the content does not contradict each other.

[0115] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. 1. An image generating means for generating a reflected wave image based on a received signal of an electromagnetic wave reflected by an object; a likelihood map generating means for generating a first likelihood map based on the intensity of the received signal shown in the reflected wave image, the first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that shields a background object and has a reflectivity of electromagnetic waves different from that of the background object, and for generating a second likelihood map based on the shape of the object shown in the reflected wave image, the second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape; An object detection device having: 2. A first identification means for identifying a region of the irregularly shaped object based on the first likelihood map; a second identification means for identifying a region of the fixed-shape object based on the second likelihood map; 1. The object detection device according to claim 1, further comprising: 3. An object detection device as described in 2, further comprising an output means for generating and outputting a first image showing the area of ​​the identified object of irregular shape and a second image showing the area of ​​the identified object of fixed shape. 4. The likelihood map generating means An object detection device described in any one of 1 to 3, which generates the first likelihood map and the second likelihood map based on a learning model that outputs both the first likelihood map and the second likelihood map in response to the input of the reflected wave image. 5. A first learning data input means for inputting, as learning data relating to the irregular-shaped object, a first learning image and first correct answer data indicating a region of the irregular-shaped object in the first learning image; a second learning data input means for inputting, as learning data relating to the fixed-shape object, a second learning image and second ground truth data indicating a region of the fixed-shape object in the second learning image; a learning means for generating the learning model based on learning data relating to the object of the irregular shape and learning data relating to the object of the fixed shape; 5. The object detection device according to claim 4, 6. The first learning data input means An object detection device as described in 5, which detects an area from the first learning image where a predetermined feature obtained from image intensity satisfies a predetermined condition as an area of ​​the irregularly shaped object, and uses the detection result as the first correct answer data. 7. The first learning data input means 7. An object detection device according to claim 5 or 6, which detects the area of ​​the irregularly shaped object in the first learning image based on the position of the irregularly shaped object identified in the image generated by the camera, and uses the detection result as the first correct answer data. 8. The second learning data input means 8. An object detection device according to any one of claims 5 to 7, which detects the area of ​​the fixed-shape object in the second learning image based on the position of the fixed-shape object identified in the image generated by a camera, and uses the detection result as the second correct answer data. 9. An object detection device according to any one of 1 to 8, wherein the object of an irregular shape includes a liquid or powdery object. 10. An object detection device according to any one of 1 to 9, wherein the background object is a human body. 11. An object detection device according to any one of 1 to 10, wherein the fixed-shape object includes a firearm, a blade, or a weapon. 12. One or more computers: A reflected wave image is generated based on the received signal of the electromagnetic wave reflected by the object. a first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that shields a background object and has a reflectivity of electromagnetic waves different from that of the background object is generated based on the intensity of the received signal shown in the reflected wave image, and a second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape is generated based on the shape of the object shown in the reflected wave image; Object detection methods. 13. Computer, an image generating means for generating a reflected wave image based on a received signal of the electromagnetic wave reflected by the object; a likelihood map generating means for generating, based on the intensity of the received signal shown in the reflected wave image, a first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that shields a background object and has a reflectivity of electromagnetic waves different from that of the background object, and for generating, based on the shape of the object shown in the reflected wave image, a second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape; A program that functions as a

[0116] This application claims priority based on Japanese Patent Application No. 2022-197899, filed December 12, 2022, the disclosure of which is incorporated herein in its entirety. [Explanation of symbols]

[0117] 10 Processing equipment 11 Image generation unit 12 Likelihood map generation unit 13 First Specific Part 14 Second specific part 15 Output section 16 Learning model generation unit 17 Learning model memory section 20 Learning Device 21 First learning data input unit 22 Second learning data input unit 23 Learning Department 1A processor 2A Memory 3A input / output I / F 4A peripheral circuit 5A Bus

Claims

1. an image generating means for generating a reflected wave image based on a received signal of the electromagnetic wave reflected by the object; a likelihood map generating means for generating a first likelihood map based on the intensity of the received signal shown in the reflected wave image, the first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that shields a background object and has a reflectivity of electromagnetic waves different from that of the background object, and for generating a second likelihood map based on the shape of the object shown in the reflected wave image, the second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape; An object detection device having:

2. a first identification means for identifying a region of the irregularly shaped object based on the first likelihood map; a second identification means for identifying a region of the fixed-shape object based on the second likelihood map; The object detection device according to claim 1 , further comprising:

3. 3. The object detection device according to claim 2, further comprising an output means for generating and outputting a first image showing the area of ​​the identified object of an irregular shape and a second image showing the area of ​​the identified object of a fixed shape.

4. The likelihood map generating means The object detection device described in claim 1, wherein the first likelihood map and the second likelihood map are generated based on a learning model that outputs both the first likelihood map and the second likelihood map in response to the input of the reflected wave image.

5. a first learning data input means for inputting, as learning data relating to the irregular-shaped object, a first learning image and first ground truth data indicating a region of the irregular-shaped object within the first learning image; a second learning data input means for inputting, as learning data relating to the fixed-shape object, a second learning image and second correct answer data indicating a region of the fixed-shape object within the second learning image; a learning means for generating the learning model based on learning data relating to the object of the irregular shape and learning data relating to the object of the fixed shape; The object detection device according to claim 4 , further comprising:

6. The first learning data input means The object detection device according to claim 5, wherein an area of ​​the first learning image where a predetermined feature obtained from image intensity satisfies a predetermined condition is detected as an area of ​​the irregularly shaped object, and the detection result is used as the first correct answer data.

7. The first learning data input means 6. The object detection device according to claim 5, wherein the area of ​​the irregularly shaped object is detected in the first learning image based on the position of the irregularly shaped object identified in the image generated by the camera, and the detection result is used as the first correct answer data.

8. The second learning data input means includes: The object detection device according to claim 5, wherein the area of ​​the fixed-shape object is detected in the second learning image based on the position of the fixed-shape object identified in the image generated by the camera, and the detection result is used as the second correct answer data.

9. One or more computers A reflected wave image is generated based on the received signal of the electromagnetic wave reflected by the object. a first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that shields a background object and has a reflectivity of electromagnetic waves different from that of the background object is generated based on the intensity of the received signal shown in the reflected wave image, and a second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape is generated based on the shape of the object shown in the reflected wave image; Object detection methods.

10. Computer, an image generating means for generating a reflected wave image based on a received signal of the electromagnetic wave reflected by the object; a likelihood map generating means for generating, based on the intensity of the received signal shown in the reflected wave image, a first likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of an irregular shape that shields a background object and has a reflectivity of electromagnetic waves different from that of the background object, and for generating, based on the shape of the object shown in the reflected wave image, a second likelihood map indicating the likelihood that each pixel of the reflected wave image corresponds to an object of a predefined fixed shape; A program that functions as a

Images