Information processing device and information processing method, and sensing system

Abstract

The information processing device pertaining to the present disclosure includes an optical transmission unit (101) that transmits light modulated by a frequency-modulated continuous wave, and an optical reception unit (103) that receives light and outputs a reception signal, and comprises: a first recognition unit (122) that performs recognition processing on the basis of a point group outputted by a light detection distance measurement unit (11) that outputs a point group including a plurality of points that each have speed information on the basis of the reception signal, and outputs three-dimensional recognition information of an object; a generating unit (125) that generates vibration distribution information indicating a vibration distribution of the object on the basis of the speed information and the three-dimensional recognition information; and a detection unit (20) that detects an anomaly in the object on the basis of the vibration distribution information.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a sensing system.

Background Art

[0002] By detecting vibrations on the surface of an object or the like, abnormalities occurring in the object can be detected. Conventionally, techniques for non-contact detection of vibrations on the surface of a measurement object or the like using optical means have been known. For example, a method of irradiating a target with laser light and detecting vibrations based on the reflected light, and a method of detecting vibrations based on an imaging image obtained by photographing a measurement object using a high-speed camera are known.

[0003] As a vibration meter using laser light, a laser Doppler vibrometer that measures vibrations using the Doppler effect is known. The laser Doppler vibrometer performs distance measurement by coherent detection on a received signal obtained by synthesizing, for example, laser light emitted as chirped light whose pulse frequency changes linearly over time and the reflected light of the emitted laser light. Such a distance measurement method using chirped light and coherent detection is called FMCW-LiDAR (Frequency Modulated Continuous Wave-Laser Imaging Detection and Ranging).

[0004] In FMCW-LiDAR, by using the Doppler effect, speed can be measured simultaneously with distance measurement. Patent Document 1 discloses a technique for performing distance measurement using laser light with continuously modulated frequency and correcting the Doppler effect during distance measurement.

[0005] Also, as one method using a high-speed camera, a method of detecting high-speed change points in a high-resolution luminance image based on a plurality of high-resolution luminance images of a measurement object and estimating vibrations in the field surface direction of the measurement object is known.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Special table 2019-537012 publication [Overview of the project] [Problems that the invention aims to solve]

[0007] A laser Doppler vibrometer can measure vibrations in the depth direction (direction of laser beam irradiation) of an object without contact. However, conventional laser Doppler vibrometers only measured vibrations at a single point on the surface of the object being measured, making it difficult to measure the vibration distribution on the surface of the object.

[0008] Furthermore, while the method using a high-speed camera allows for estimation of vibrations in the field of view of the object being measured, it was difficult to detect vibrations in the depth direction of the object being measured.

[0009] This disclosure provides an information processing device and information processing method, as well as a sensing system, that can non-contact detect abnormalities in an object being measured based on vibration distribution in the depth direction and the field of view direction. [Means for solving the problem]

[0010] The information processing apparatus according to this disclosure includes an optical transmitting unit that transmits light modulated by a frequency continuous modulated wave, and an optical receiving unit that receives light and outputs a received signal. The apparatus also includes a first recognition unit that performs recognition processing based on the point cloud output by an optical detection distance measuring unit that outputs a point cloud containing a plurality of points, each having velocity information, based on the received signal, and outputs three-dimensional recognition information of an object, a generation unit that generates vibration distribution information showing the vibration distribution of the object based on the velocity information and the three-dimensional recognition information, and a detection unit that performs abnormality detection of the object based on the vibration distribution information.

[0011] The information processing method according to this disclosure includes an optical transmitting unit that transmits light modulated by a frequency continuous modulated wave, and an optical receiving unit that receives light and outputs a received signal, and comprises a first recognition step of performing recognition processing based on the point cloud output by an optical detection distance measuring unit that outputs a point cloud including a plurality of points each having velocity information based on the received signal, and outputting three-dimensional recognition information of an object, a generation step of generating vibration distribution information showing the vibration distribution of the object based on the velocity information and the three-dimensional recognition information, and a detection step of detecting anomalies in the object based on the vibration distribution information.

[0012] The sensing system according to this disclosure includes an optical transmitting unit that transmits light modulated by a frequency continuous modulated wave, and an optical receiving unit that receives light and outputs a received signal, and comprises an optical detection distance measuring unit that outputs a point cloud including a plurality of points each having velocity information based on the received signal, a first recognition unit that performs recognition processing based on the point cloud output by the optical detection distance measuring unit and outputs three-dimensional recognition information of an object, a generation unit that generates vibration distribution information indicating the vibration distribution of the object based on the velocity information and the three-dimensional recognition information, and a detection unit that performs abnormality detection of the object based on the vibration distribution information. [Brief explanation of the drawing]

[0013] [Figure 1] This is a schematic diagram illustrating the detection of an object's velocity using optical means based on existing technology. [Figure 2] This is a schematic diagram illustrating an example of detecting the velocity of an object using an optical detection rangefinder that performs distance measurement using FMCW-LiDAR, based on existing technology. [Figure 3] This is a schematic diagram illustrating velocity detection in the field of view using FMCW-LiDAR, applicable to this disclosure. [Figure 4] This is a schematic diagram illustrating the sensing system related to this disclosure. [Figure 5] This diagram shows an example configuration of an optical detection and ranging unit applicable to each embodiment of the present disclosure. [Figure 6]It is a schematic diagram schematically showing an example of scanning of transmitted light by a scanning unit. [Figure 7] It is a block diagram showing a configuration example of an information processing apparatus applicable to each embodiment of the present disclosure. [Figure 8] It is a block diagram showing in more detail a configuration example of a sensing system according to the present disclosure. [Figure 9] It is a flowchart showing an example of an abnormality detection process according to the first embodiment. [Figure 10] It is a schematic diagram showing an example of an image displayed on a display unit in the abnormality detection process according to the first embodiment. [[ID=!3]] [Figure 11] It is a flowchart showing an example of a vibration distribution generation process according to the first embodiment. [Figure 12] It is a block diagram showing a configuration example of a sensing system according to the second embodiment. [Figure 13] It is a flowchart showing an example of an abnormality detection process according to the second embodiment. [Figure 14] It is a schematic diagram showing an example of an image displayed on a display unit in the abnormality detection process according to the second embodiment. [Figure 15] It is a flowchart showing an example of a vibration distribution generation process according to the second embodiment. [Figure 16] It is a block diagram showing a configuration example of a sensing system according to the third embodiment. [Figure 17] It is a flowchart showing an example of an abnormality detection process according to the third embodiment. [Figure 18] It is a schematic diagram showing an example of an image displayed on a display unit in the abnormality detection process according to the third embodiment. [Figure 19] It is a flowchart showing an example of a vibration distribution generation process according to the third embodiment.

MODE FOR CARRYING OUT THE INVENTION

[0014] The embodiments of this disclosure will be described in detail below with reference to the drawings. In the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0015] The embodiments of this disclosure will be described below in the following order. 1. Regarding existing technologies 2. Outline of this disclosure 2-1. Technologies applicable to this disclosure 2-2. Overview of the Sensing System Related to This Disclosure 3. First Embodiment 4. Second Embodiment 5. Third Embodiment

[0016] (1. Regarding existing technologies) This disclosure relates to a suitable technique for anomaly detection based on the vibration distribution of an object. Prior to describing each embodiment of this disclosure, existing techniques related to the techniques of this disclosure will be briefly described for the sake of understanding.

[0017] Conventionally, there are known techniques for detecting the velocity of an object using optical means and detecting the vibration of the object based on the detected velocity. Figure 1 is a schematic diagram illustrating the detection of object velocity using optical means according to existing technology.

[0018] Section (a) of Figure 1 shows an example of detecting the velocity of an object 710 using a laser Doppler velocometer 700. For example, the laser Doppler velocometer 700 shines a laser beam 720 onto a moving object 710 and receives the reflected light 730. The laser Doppler velocometer 700 can measure the velocity of the object 710 in the direction of the optical axis of the laser beam 720 by calculating the Doppler shift amount of the received reflected light 730. By repeating this measurement at predetermined time intervals, the velocity displacement of the object 710 can be determined, and based on this velocity displacement, the vibration of the object 710 in the direction of the optical axis of the laser beam 720 is detected.

[0019] Section (b) of Figure 1 shows an example of detecting the velocity of an object 710 using a Doppler radar 701 that utilizes millimeter waves 721. Similar to the laser Doppler velocometer 700 described above, the Doppler radar 7001 irradiates the moving object 710 with millimeter waves 721 and receives the reflected waves 731. By calculating the Doppler shift amount of the received reflected waves 731, the Doppler radar 701 can measure the velocity of the object 710 in the direction of irradiation of the millimeter waves 721.

[0020] Section (c) of Figure 1 shows an example of detecting the velocity of an object 710 using a high-speed camera 702. The high-speed camera 702 images the object 710 by high-speed continuous shooting and obtains multiple captured images 722. Based on edge information and patterns in these multiple captured images 722, the velocity of the object 710 in the field of view of the high-speed camera 702 can be detected, and vibrations of the object 710 in that field of view can be detected based on the velocity displacement.

[0021] (2. Outline of this disclosure) Next, I will give a brief overview of this disclosure.

[0022] (2-1. Technologies applicable to this disclosure) Figure 2 is a schematic diagram showing an example of detecting the velocity of an object 710 using an optical detection rangefinder 703 that performs distance measurement using FMCW-LiDAR, which is applicable to this disclosure.

[0023] FMCW-LiDAR (Frequency Modulated Continuous Wave-Laser Imaging Detection and Ranging) uses chirp light, which is a laser beam whose pulse frequency is changed linearly over time, as the emitted laser light. In FMCW-LiDAR, distance measurement is performed by coherent detection on a received signal which is a combination of the emitted chirp light and the reflected light of the emitted laser beam. Furthermore, FMCW-LiDAR can measure speed simultaneously with distance measurement by utilizing the Doppler effect.

[0024] In Figure 2, the optical distance measuring device 703 scans a chirp laser beam 750 within a predetermined scanning range 740, acquiring measurement values ​​at each point 741. The measurement values ​​include three-dimensional position information (3D coordinates), velocity information, and brightness information. The optical distance measuring device 703 outputs the measurement values ​​of each point 741 within the scanning range 740 as a point cloud. That is, the point cloud is a collection of points containing 3D coordinates, velocity information, and brightness information.

[0025] The optical distance measuring device 703 allows for the acquisition of continuous changes in velocity and 3D coordinates by intensively scanning the target region including the object 710, and enables the measurement of velocity, acceleration, frequency per unit time, displacement, etc., for each point included in the point cloud. Based on the information measured from this target region, the vibration distribution in the target region can be determined.

[0026] Furthermore, FMCW-LiDAR can detect not only the velocity of the object 710 in the depth direction (along the optical axis direction of the laser beam) but also the velocity in the field of view direction by analyzing the velocity distribution of the point cloud. The field of view direction refers to the direction of the plane that intersects the optical axis direction of the laser beam emitted from the optical detection rangefinder 703 at an emission angle of 0°. The field of view direction may, for example, be the direction of the plane that intersects the optical axis direction at a right angle.

[0027] Figure 3 is a schematic diagram illustrating velocity detection in the field of view using FMCW-LiDAR, applicable to this disclosure. Section (a) of Figure 3 shows an example where the object 710 is moving to the left in the field of view of the optical rangefinder 703, and section (b) shows an example where the object 710 is moving downward in a direction perpendicular to the field of view of the optical rangefinder 703.

[0028] In the example in section (a) of Figure 3, the point 741 on the left side of the object 710 is moving away from the optical rangefinder 703, so the velocity 760 takes a positive value, for example. On the other hand, the point 741 on the right side of the object 710 is moving towards the optical rangefinder 703, so the velocity 760 takes a negative value, for example. The absolute values ​​of these velocities 760 increase as the position of point 741 moves away from the position corresponding to the optical rangefinder 703. In the example in section (b) of Figure 3, the velocity 760 changes depending on the distance of each point 741 on the object 710 from the optical rangefinder 703.

[0029] By analyzing the velocity distribution of the point cloud from each of these 741 points, the velocity in the field of view can be measured. Conventional image sensors, for example, that can acquire color information for each color (R (red), G (green), and B (blue)), can measure the velocity in the field of view by extracting feature points such as edges and patterns in the captured image and detecting the frame difference of the extracted feature points. In contrast, FMCW-LiDAR can acquire velocity and vibration distribution even for points that do not have feature points.

[0030] (2-2. Overview of the sensing system related to this disclosure) Next, we will explain the overview of the sensing system related to this disclosure.

[0031] Figure 4 is a schematic diagram illustrating the sensing system according to this disclosure. In Figure 4, the sensing system 1 includes a sensor unit 10, a signal processing unit 12, and an anomaly detection unit 20.

[0032] The sensor unit 10 includes an optical detection distance measuring unit 11 that measures the distance to the object 50 being measured using FMCW-LiDAR. The optical detection distance measuring unit 11 has a laser beam scanning mechanism such as a mechanical scanner, MEMS (Micro Electro Mechanical Systems), or OPA (Optical Phased Array). The optical detection distance measuring unit 11 scans a laser beam generated by chirp light according to a scan line 41 within a predetermined scan range 40 and acquires a point cloud containing information about each measurement point along the scan line 41. The point cloud includes velocity information indicating the velocity 60 at each point on the object 50, the 3D coordinates of each point, etc. The optical detection distance measuring unit 11 acquires a point cloud for one frame (referred to as a frame point cloud) with one scan of the scan range 40.

[0033] The signal processing unit 12 performs signal processing on the point cloud acquired by the optical detection distance measurement unit 11 to obtain vibration distribution information showing the vibration distribution in the object 50. The anomaly detection unit 20 detects whether or not there is an anomaly in the object 50 based on the vibration distribution information of the object 50 acquired by the signal processing unit 12.

[0034] In sensing system 1, the signal processing unit 12 and the anomaly detection unit 20 may be configured by executing an information processing program on an information processing device having a CPU (Central Processing Unit), for example. However, the system is not limited to this, and one or both of the signal processing unit 12 and the anomaly detection unit 20 may be configured as hardware devices, or the signal processing unit 12 and the anomaly detection unit 20 may be configured on different information processing devices.

[0035] Figure 5 is a block diagram showing an example configuration of an optical detection distance measuring unit 11 applicable to each embodiment of the present disclosure. In Figure 5, the optical detection distance measuring unit 11 includes a scanning unit 100, an optical transmission unit 101, an optical reception unit 103, a first control unit 110, a second control unit 115, a point cloud generation unit 130, a pre-processing unit 140, and an interface (I / F) unit 141.

[0036] Furthermore, the first control unit 110 includes a scanning control unit 111 and an angle detection unit 112, and controls scanning by the scanning unit 100. The second control unit 115 includes a transmitted light control unit 116 and a received signal processing unit 117, and controls the transmission of laser light by the optical detection distance measuring unit 11 and processes the received light.

[0037] The optical transmitting unit 101 includes, for example, a light source such as a laser diode for emitting laser light which is the transmitted light, an optical system for emitting the light emitted by the light source, and a laser output modulator for driving the light source. The optical transmitting unit 101 causes the light source to emit light in response to an optical transmission control signal supplied from the optical transmission control unit 116, which will be described later, and emits transmitted light as chirp light whose frequency changes linearly within a predetermined frequency range as time progresses. The transmitted light is sent to the scanning unit 100 and also to the optical receiving unit 103 as local emission.

[0038] The transmitting optical control unit 116 generates a signal whose frequency changes linearly (for example, increases) within a predetermined frequency range as time progresses. Such a signal whose frequency changes linearly within a predetermined frequency range as time progresses is called a chirp signal. Based on this chirp signal, the transmitting optical control unit 116 generates a modulation synchronization timing signal that is input to the laser output modulation device included in the optical transmitting unit 101. The transmitting optical control unit 116 supplies the generated optical transmission control signal to the optical transmitting unit 101 and the point cloud generation unit 130.

[0039] The received light received by the scanning unit 100 is input to the optical receiving unit 103. The optical receiving unit 103 includes, for example, a light receiving unit that receives (receives) the input received light and a drive circuit that drives the light receiving unit. The light receiving unit can be, for example, a pixel array in which light-receiving elements such as photodiodes, each constituting a pixel, are arranged in a two-dimensional grid.

[0040] The optical receiving unit 103 further includes a combining unit that combines the input received light and the local light emitted from the optical transmitting unit 101. If the received light is reflected light from the object to which the transmitted light is directed, the received light becomes a signal that is delayed relative to the local light emitted according to the distance to the object, and each combined signal obtained by combining the received light and the local light emitted becomes a signal of a constant frequency (beat signal). The optical receiving unit 103 supplies the signal corresponding to the received light as a received signal to the received signal processing unit 117.

[0041] The received signal processing unit 117 performs signal processing on the received signal supplied from the optical receiver 103, such as a fast Fourier transform. Through this signal processing, the received signal processing unit 117 determines the distance to the object and the velocity of the object, and generates measurement information including distance information and velocity information, respectively. The received signal processing unit 117 further determines brightness information indicating the brightness of the object based on the received signal and includes it in the measurement information. The received signal processing unit 117 supplies the generated measurement information to the point cloud generation unit 130.

[0042] The scanning unit 100 transmits the transmitted light sent from the optical transmission unit 101 at an angle according to the scanning control signal supplied from the scanning control unit 111, and receives the light incident from that angle as received light. In the scanning unit 100, if, for example, a two-axis mirror scanning device is used as the scanning mechanism for the transmitted light, the scanning control signal becomes, for example, a drive voltage signal applied to each axis of the two-axis mirror scanning device.

[0043] The scanning control unit 111 generates a scanning control signal that changes the transmission and reception angle of the scanning unit 100 within a predetermined angular range, and supplies it to the scanning unit 100. The scanning unit 100 can perform scanning of a certain range with transmitted light according to the supplied scanning control signal.

[0044] The scanning unit 100 has a sensor that detects the emission angle of the transmitted light to be emitted, and outputs an angle detection signal indicating the emission angle of the transmitted light detected by this sensor. The angle detection unit 112 determines the transmission and reception angles based on the angle detection signal output from the scanning unit 100, and generates angle information indicating the determined angles. The angle detection unit 112 supplies the generated angle information to the point cloud generation unit 130.

[0045] Figure 6 is a schematic diagram illustrating an example of scanning of transmitted light by the scanning unit 100. The scanning unit 100 scans within a scanning range 40 corresponding to a predetermined angular range, following a predetermined number of scan lines 41. Each scan line 41 corresponds to a single trajectory scanned between the left and right ends of the scanning range 40. In response to a scanning control signal, the scanning unit 100 scans between the upper and lower ends of the scanning range 40 according to the scan lines 41.

[0046] At this time, the scanning unit 100, in accordance with the scanning control signal, sequentially and discretely changes the chirp light emission points along the scan line 41 at a constant time interval (point rate), for example, points 2201, 2202, 2203, ... At this time, the scanning speed by the two-axis mirror scanning device slows down near the turning points at the left and right ends of the scanning range 40 of the scan line 41. Therefore, the points 2201, 2202, 2203, ... will not be arranged in a grid pattern within the scanning range 40. The optical transmitting unit 101 may emit chirp light one or more times for a single emission point in accordance with the optical transmission control signal supplied from the transmitting optical control unit 116.

[0047] Returning to the explanation of Figure 5, the point cloud generation unit 130 generates a point cloud based on angle information supplied from the angle detection unit 112, optical transmission control signals supplied from the transmission optical control unit 116, and measurement information supplied from the reception signal processing unit 113. More specifically, the point cloud generation unit 130 identifies a point in space based on the angle and distance, using the angle information and distance information included in the measurement information. The point cloud generation unit 130 obtains a point cloud as a collection of the identified points under predetermined conditions. Based on the velocity information and brightness information included in the measurement information, the point cloud generation unit 130 calculates the point cloud by taking into account the velocity and brightness of each identified point. That is, the point cloud includes information indicating the 3D coordinates, velocity, and brightness for each point included in the point cloud.

[0048] The point cloud generation unit 130 supplies the generated point cloud to the pre-processing unit 140. The pre-processing unit 140 performs predetermined signal processing, such as format conversion, on the supplied point cloud. The point cloud processed by the pre-processing unit 140 is output to the outside of the optical detection distance measuring unit 11 via the I / F unit 141. The point cloud output from the I / F unit 141 includes 3D coordinate information, velocity information, and brightness information for each point included in the point cloud.

[0049] Figure 7 is a block diagram showing the configuration of an example of an information processing device applicable to each embodiment of the present disclosure. In Figure 7, the information processing device 1000 includes a CPU 1010, a ROM (Read Only Memory) 1011, a RAM (Random Access Memory) 1012, a display unit 1013, a storage device 1014, an input device 1015, a communication interface (I / F) 1016, and a sensor unit I / F 1017, all of which are connected to each other via a bus 1020 so as to be able to communicate with each other.

[0050] The storage device 1014 can be a hard disk drive or non-volatile memory (flash memory), and stores various programs and data. The CPU 1010 controls the operation of the entire information processing device 1000 using the RAM 1012 as work memory, according to the programs stored in the ROM 1011 and the storage device 1014.

[0051] The display unit 1013 includes a display control unit that generates display signals based on display control information generated by the CPU 1010, and a display device that performs display according to the display signals generated by the display control unit. The display device may be a display device that is externally connected to and used with respect to the information processing device 1000.

[0052] The input device 1015 accepts user input, such as from a keyboard. Information corresponding to the user input received by the input device 1015 is passed to the CPU 1010. The input device 1015 may also be a touch panel integrated with the display device included in the display unit 1013.

[0053] The communication interface 1016 is an interface for the information processing device 1000 to communicate with external devices. Communication via the communication interface 1016 may be via a network, or it may be through a direct connection to hardware devices or the like to the information processing device 1000. Furthermore, communication via the communication interface 1016 may be wired or wireless.

[0054] The sensor unit I / F 1017 is an interface for connecting the sensor unit 10. In the sensor unit 10, the point cloud output from the optical detection distance measuring unit 11 is passed to the CPU 1010 via the sensor unit I / F 1017.

[0055] In the information processing device 1000, the CPU 1010 executes an information processing program for realizing the sensing system 1 according to this disclosure, thereby configuring the above-mentioned signal processing unit 12 and anomaly detection unit 20 as modules, for example, on the main memory area of ​​the RAM 1012.

[0056] The information processing program can be obtained from an external source (for example, another server device not shown) via communication via the communication interface 1016, and installed on the information processing device 1000. However, the information processing program may also be provided stored on a removable storage medium such as a CD (Compact Disk), DVD (Digital Versatile Disk), or USB (Universal Serial Bus) memory.

[0057] In this example, the signal processing unit 12 and the anomaly detection unit 20 are configured on the same information processing device 1000, but this is not limited to this example. The signal processing unit 12 and the anomaly detection unit 20 may be configured on different hardware (such as information processing devices).

[0058] (3. First Embodiment) Next, a first embodiment of this disclosure will be described.

[0059] Figure 8 is a block diagram showing in more detail the configuration of an example of a sensing system according to this disclosure. In Figure 8, the sensing system 1a includes a sensor unit 10, a signal processing unit 12a, and an anomaly detection unit 20. The sensor unit 10 includes an optical detection distance measuring unit 11 and a signal processing unit 12a. The signal processing unit 12a includes a 3D object detection unit 121, a 3D object recognition unit 122, an I / F unit 123, a vibration distribution generation unit 125, and a storage unit 126.

[0060] These 3D object detection unit 121, 3D object recognition unit 122, I / F unit 123, and vibration distribution generation unit 125 can be configured, for example, by executing an information processing program according to this disclosure on the CPU 1010 of an information processing device 1000. However, some or all of these 3D object detection unit 121, 3D object recognition unit 122, I / F unit 123, and vibration distribution generation unit 125 may be configured by hardware circuits that work together.

[0061] The point cloud output from the optical detection distance measurement unit 11 is input to the signal processing unit 12a, which then supplies it to the I / F unit 123 and the 3D object detection unit 121.

[0062] The 3D object detection unit 121 detects measurement points that indicate 3D objects within the supplied point cloud. To avoid complexity, expressions such as "detects measurement points that indicate 3D objects within the point cloud" will be replaced with expressions such as "detects 3D objects within the point cloud."

[0063] The 3D object detection unit 121 detects point clouds that have velocity, and point clouds that include the said point clouds and have relationships such as having a certain density of connections or higher, as point clouds corresponding to 3D objects (called localized point clouds). For example, in order to distinguish between static and dynamic objects included in the point cloud, the 3D object detection unit 121 extracts points from the point cloud that have a certain absolute velocity value or higher. From the point clouds formed by the extracted points, the 3D object detection unit 121 detects a set of point clouds that are localized within a certain spatial range (corresponding to the size of the target object) as a localized point cloud corresponding to a 3D object. The 3D object detection unit 121 may extract multiple localized point clouds from the point cloud.

[0064] The 3D object detection unit 121 acquires the 3D coordinates, velocity information, and brightness information for each point in the detected localized point group. The 3D object detection unit 121 outputs the 3D coordinates, velocity information, and brightness information for these localized point groups as 3D detection information indicating the 3D detection result. The 3D object detection unit 121 may also add label information indicating the 3D object corresponding to the detected localized point group to the region of the localized point group and include the added label information in the 3D detection result.

[0065] The 3D object recognition unit 122 acquires 3D detection information output from the 3D object detection unit 121. Based on the acquired 3D detection information, the 3D object recognition unit 122 performs object recognition on the localized point cloud indicated by the 3D detection information. For example, if the number of points included in the localized point cloud indicated by the 3D detection information is greater than or equal to a predetermined number that can be used to recognize the target object, the 3D object recognition unit 122 performs point cloud recognition processing on that localized point cloud. Through this point cloud recognition processing, the 3D object recognition unit 122 estimates attribute information about the recognized object.

[0066] The 3D object recognition unit 122 performs object recognition processing on the localized point cloud corresponding to the 3D object from the point cloud output from the optical detection distance measurement unit 11. For example, the 3D object recognition unit 122 removes the point cloud portion other than the localized point cloud from the point cloud output from the optical detection distance measurement unit 11 and does not perform object recognition processing on that portion. This makes it possible to reduce the processing load on the recognition unit 122.

[0067] The 3D object recognition unit 122 outputs the recognition result for the localized point cloud as 3D recognition information when the confidence level of the estimated attribute information is above a certain level, that is, when the recognition process can be performed successfully. The 3D object recognition unit 122 can include the 3D coordinates, 3D size, velocity information, attribute information, and confidence level of the localized point cloud in the 3D recognition information. The attribute information is information that indicates the attributes of the target object, such as the type of target object to which the unit belongs, for each point in the point cloud as a result of the recognition process.

[0068] The 3D recognition information output from the 3D object recognition unit 122 is input to the I / F unit 123. As described above, the point cloud output from the optical detection distance measurement unit 11 is also input to the I / F unit 123. The I / F unit 123 integrates the point cloud with the 3D recognition information and supplies it to the vibration distribution generation unit 125. The I / F unit 123 also supplies the point cloud supplied from the optical detection distance measurement unit 11 to the anomaly detection unit 20, which will be described later.

[0069] The vibration distribution generation unit 125 estimates the vibration distribution in the object 50 and generates vibration distribution information based on the point cloud and 3D recognition information supplied from the I / F unit 123. The vibration distribution generation unit 125 may estimate the vibration distribution of the object 50 using the supplied 3D recognition information and past 3D recognition information related to the localized point cloud stored in the storage unit 126.

[0070] The vibration distribution generation unit 125 supplies vibration distribution information, which represents the estimated vibration distribution, to the anomaly detection unit 20. The vibration distribution generation unit 125 also stores the point cloud (localized point cloud) and 3D recognition information as historical information in the memory unit 126.

[0071] Furthermore, the vibration distribution generation unit 125 can generate display control information for displaying an image to be presented to the user, based on the point cloud and 3D recognition information supplied from the I / F unit 123.

[0072] The anomaly detection unit 20 detects anomalies in the object 50 based on the point cloud supplied from the signal processing unit 12a and the vibration distribution information. For example, the anomaly detection unit 20 may generate an evaluation value based on the vibration distribution information and determine whether or not there is an anomaly in the object 50 by performing a threshold judgment on the generated evaluation value. The anomaly detection unit 20 outputs the anomaly detection result for the object 50 to an external source, for example.

[0073] Figure 9 is a flowchart illustrating an example of the anomaly detection process according to the first embodiment. Figure 10 is a schematic diagram showing an example of an image displayed on the display unit 1013 in the anomaly detection process according to the first embodiment.

[0074] In Figure 9, in step S10, the sensing system 1a scans the entire area of ​​the scanning range 40 using the optical detection distance measuring unit 11 of the sensor unit 10 and acquires a point cloud within the scanning range 40. In the next step S11, the sensing system 1a generates a two-dimensional (2D) image of the scanning range 40 based on the point cloud acquired in the scan of step S10 using the vibration distribution generation unit 125, and generates display control information for displaying the 2D image in 2D display mode. Since the point cloud only contains brightness information for display purposes, the 2D image displayed in this 2D display mode is a monochrome image. This 2D image is displayed, for example, on the display unit 1013 of the information processing device 1000.

[0075] In the next step, S12, the sensing system 1a determines whether or not an ROI (Region of Interest) has been set for the point cloud acquired in step S10 within the scanning range 40.

[0076] For example, in step S11, the sensing system 1a sets an ROI in response to user operation on the 2D image displayed by the display unit 1013. If the vibration distribution generation unit 125 determines that no ROI has been set (step S12, "No"), the sensing system 1a returns to step S12. On the other hand, if the vibration distribution generation unit 125 determines that an ROI has been set (step S12, "Yes"), the sensing system 1a proceeds to step S13.

[0077] Image 300a, shown in the upper left of Figure 10, shows an example of a 2D image in which an ROI 301 is defined by a rectangle. Image 300a is a 2D image based on a point cloud acquired by scanning the scanning range 40 with the optical detection range measuring unit 11. Therefore, image 300a has a resolution corresponding to the points 2201, 2202, 2203, ... (see Figure 6) from which the laser beam is emitted by the optical detection range measuring unit 11 within the scanning range 40.

[0078] In Figure 9, in step S13, the sensing system 1a presents the set ROI 301 using the vibration distribution generation unit 125. More specifically, the vibration distribution generation unit 125 displays the set ROI 301 on the display unit 1013 in 3D display mode. In 3D display mode, the vibration distribution generation unit 125 displays candidate targets for detecting vibration distributions included in the ROI 301, based on the 3D recognition information, which is the object recognition result from the 3D object recognition unit 122.

[0079] Image 300b, shown in the upper right of Figure 10, shows an example of an enlarged view of ROI 301 in Image 300a in the upper left of the same figure, along with the display of target candidates. In the example of Image 300b, each of the regions 310a to 310e, which were extracted as candidate target regions based on 3D recognition information, is shown with diagonal lines. These regions 310a to 310e were recognized by the 3D object recognition unit 122, for example, as different parts within ROI 301.

[0080] In Figure 9, in the next step S14, the sensing system 1a determines whether a target for vibration distribution detection has been selected from each of the regions 310a to 310e presented as candidate target regions in step S13, using the vibration distribution generation unit 125.

[0081] For example, in step S13, the sensing system 1a selects a target area for vibration distribution detection from each of the areas 310a to 310e in response to user operation on the image 300b displayed by the display unit 1013. If the vibration distribution generation unit 125 determines that no target area has been set (step S14, "No"), the sensing system 1a returns to step S14. On the other hand, if the vibration distribution generation unit 125 determines that a target area has been selected (step S14, "Yes"), the sensing system 1a proceeds to step S15.

[0082] Image 300c, shown in the lower left of Figure 10, illustrates an example where region 310b is selected as the target region from among the candidate target regions in step S14. For illustrative purposes, only region 310b is selected as the target region for vibration distribution detection from multiple candidate regions 310a to 310e; however, this is not limited to this example, and it is possible to select multiple regions as the target region.

[0083] In step S15, the sensing system 1a uses the vibration distribution generation unit 125 to detect the vibration distribution for the target region selected in step S14 (region 310b in this example) and outputs the vibration distribution for the target region. The vibration distribution generation process by the vibration distribution generation unit 125 will be described later.

[0084] Image 300d, shown in the lower right of Figure 10, is a schematic image of the vibration distribution in the target region (region 310b in this example), generated by the vibration distribution generation unit 125. In image 300d, the target region 310b is divided into regions 320a to 320d according to the degree of vibration, and the vibration distribution in region 310b is shown. Image 300d may be displayed on the display unit 1013 and presented to the user, or it may be information held internally by the vibration distribution generation unit 125.

[0085] Vibration parameters that indicate the degree of vibration include, for example, frequency F (Hz), displacement D (mm), velocity v (m / s), and acceleration A (m / s). 2 ) may be the case. The vibration distribution may be expressed using, for example, one of the values ​​of frequency F, displacement D, velocity v, and acceleration A as a representative value, or it may be expressed as a value that combines two or more of these vibration parameters. It is not limited to this, for example, the distribution may be determined for each of these vibration parameters.

[0086] In the next step S16, the sensing system 1a, using the anomaly detection unit 20, determines whether there is a region in the target area where the degree of vibration exceeds a threshold, based on the vibration distribution output from the vibration distribution generation unit 125. The vibration distribution generation unit 125 performs a threshold determination on the degree of vibration as described above, for example.

[0087] If the anomaly detection unit 20 determines that there is a region within the target region where the degree of vibration exceeds a threshold (step S16, "Yes"), the sensing system 1a proceeds to step S17 and determines that an anomaly has been detected in the target region. On the other hand, if the anomaly detection unit 20 determines that there is no region within the target region where the degree of vibration exceeds a threshold (step S16, "No"), the sensing system 1a proceeds to step S18 and determines that there is no anomaly in the target region.

[0088] After the processing in step S17 or step S18, the series of processes shown in the flowchart in Figure 9 is completed.

[0089] Figure 11 is a flowchart illustrating an example of the vibration distribution generation process according to the first embodiment. The flowchart shown in Figure 11 is an example that shows the process of step S15 in Figure 9 described above in more detail, and is executed, for example, in the vibration distribution generation unit 125.

[0090] In Figure 11, the processes in steps S100 to S104 on the left measure the vibration distribution in the depth direction and the field of view direction using the velocity information of the point cloud. The processes in steps S110 to S114 on the right measure the vibration distribution in the field of view direction using the luminance information of the point cloud. Hereafter, when velocity information is used, the point cloud will be appropriately referred to as the velocity point cloud, and when luminance information is used, the point cloud will be appropriately referred to as the luminance point cloud.

[0091] First, we will explain the vibration distribution measurement process in the depth direction and field of view direction using velocity point clouds, from step S100 to step S104.

[0092] In Figure 11, in step S100, the vibration distribution generation unit 125 acquires a point cloud frame using velocity point clouds over the entire scanning range 40 based on the output of the optical detection distance measurement unit 11. In the next step S101, the vibration distribution generation unit 125 acquires 3D recognition information obtained by the recognition processing of this point cloud frame using velocity point clouds by the 3D object recognition unit 122. This 3D recognition information includes the 3D position, 3D size, velocity, attributes, and confidence level of the object recognized from the point cloud frame using velocity point clouds.

[0093] In the next step S102, the vibration distribution generation unit 125 extracts the velocity point cloud of the target region from the point cloud frame. The velocity point cloud extracted in step S102 contains 3D information. The vibration distribution generation unit 125 may store the extracted velocity point cloud of the target region in the storage unit 126.

[0094] In the next step, S103, the vibration distribution generation unit 125 determines whether or not it has performed measurements on a predetermined number of point cloud frames necessary for detecting vibrations in the depth direction. If the vibration distribution generation unit 125 determines that it has not performed measurements on a predetermined number of point cloud frames (step S103, "No"), it returns to step S100 to acquire the next point cloud frame and perform measurements on the acquired point cloud frame (steps S101, S102).

[0095] On the other hand, if the vibration distribution generation unit 125 determines that it has performed measurements of a predetermined number of point cloud frames (step S103, "Yes"), it proceeds to step S104.

[0096] In step S104, the vibration distribution generation unit 125 calculates the vibration distribution in the depth direction (see Figure 2) and the vibration distribution in the field of view direction (see Figure 3) based on the velocity point cloud obtained in the processing up to step S103, using, for example, the method described with reference to Figures 2 and 3.

[0097] Next, we will explain the vibration distribution measurement process in the field of view direction using the luminance point cloud, which takes place in steps S110 to S114.

[0098] In Figure 11, in step S110, the vibration distribution generation unit 125 acquires a point cloud frame of luminance points over the entire scanning range 40 based on the output of the optical detection distance measurement unit 11. In the next step S111, the vibration distribution generation unit 125 acquires 3D recognition information obtained by the recognition processing of this point cloud frame of luminance points by the 3D object recognition unit 122. This 3D recognition information includes the 3D position, 3D size, velocity, attributes, and confidence level of the object recognized from the point cloud frame of luminance points. In the next step S112, the vibration distribution generation unit 125 extracts the luminance point cloud of the target region from the point cloud frame. The vibration distribution generation unit 125 may store the extracted luminance point cloud of the target region in the storage unit 126.

[0099] The vibration distribution generation unit 125 acquires 2D information from a point cloud frame composed of luminance point clouds. For example, the vibration distribution generation unit 125 projects the information of each point included in the point cloud frame onto a plane in the field of view direction. Therefore, the luminance point cloud extracted in step S112 has 2D information.

[0100] In the next step, S113, the vibration distribution generation unit 125 determines whether or not it has performed measurements on a predetermined number of point cloud frames necessary for detecting vibrations in the field of view direction. If the vibration distribution generation unit 125 determines that it has not performed measurements on a predetermined number of point cloud frames (step S113, "No"), it returns to step S100 to acquire the next point cloud frame and perform measurements on the acquired point cloud frame (steps S111, S112).

[0101] On the other hand, if the vibration distribution generation unit 125 determines that it has performed measurements of a predetermined number of point cloud frames (step S113, "Yes"), it proceeds to step S114.

[0102] In step S114, the vibration distribution generation unit 125 calculates the vibration distribution in the field of view direction based on the luminance point clouds of multiple frames, each of which is 2D information, acquired in the processing up to step S113, for example, using the method described using section (c) of Figure 1.

[0103] After completing the processing in steps S104 and S114, the vibration distribution generation unit 125 moves the processing to step S120. In step S120, the vibration distribution generation unit 125 integrates the vibration distributions in the depth direction and field of view direction calculated in step S104 with the vibration distribution in the field of view direction calculated in step S114 to obtain the vibration distribution of the target area, and outputs vibration distribution information showing the obtained vibration distribution to the anomaly detection unit 20.

[0104] In the flowchart of Figure 11, the processes in step S100 and step S110 can be executed independently of each other. However, the processes in steps S100 and S110 may be executed synchronously. Furthermore, steps S100 and S110 may be processes performed by scanning the same scanning range 40. In this case, the predetermined numbers determined in steps S103 and S113 will be the same numbers as in steps S101 and S113.

[0105] Thus, in the first embodiment, 3D object recognition processing is performed on the point cloud output by the optical detection ranging unit 11, which measures distance using FMCW-LiDAR, and the point cloud of the target region is extracted based on the recognition result. Therefore, it is possible to measure the vibration distribution within the target region, and abnormalities in the object can be detected based on the measured vibration distribution.

[0106] (4. Second Embodiment) Next, a second embodiment of the present disclosure will be described. In the second embodiment, a target region is further set for the ROI, and the scanning range of the optical detection distance measuring unit 11 is limited to this target region. By limiting the scanning range in this way, the resolution within the scanning range can be made variable, making it possible to detect vibration distributions even for objects at a distance, for example.

[0107] Figure 12 is a block diagram showing an example configuration of a sensing system according to the second embodiment. Note that detailed explanations of parts common to Figure 8 described above will be omitted below.

[0108] In Figure 12, the sensing system 1b according to the second embodiment includes a sensor unit 10a, a signal processing unit 12b, and an anomaly detection unit 20. The sensor unit 10a includes an optical detection distance measuring unit 11a. The optical detection distance measuring unit 11a has the same configuration as the optical detection distance measuring unit 11 described with reference to Figure 5, and is capable of controlling the scanning range in which the scanning unit 100 performs scanning in accordance with a local scanning control signal supplied from the outside, in addition to a scanning control signal generated internally. For example, the scanning unit 100 is capable of intensively scanning any area within the scanning range 40 of the entire optical detection distance measuring unit 11a as a new scanning range in accordance with the local scanning control signal.

[0109] The signal processing unit 12b has a configuration that adds a local scanning control unit 170 to the signal processing unit 12a in the sensing system 1a according to the first embodiment shown in Figure 8. Based on the target area setting information and the 3D recognition information output from the 3D object recognition unit 122, the local scanning control unit 170 sets a target area, which is a narrower scanning range than the entire scanning range 40 of the optical detection distance measurement unit 11, within the scanning range 40. The local scanning control unit 170 outputs a local scanning control signal for scanning the set target area.

[0110] Figure 13 is a flowchart illustrating an example of the anomaly detection process according to the second embodiment. Figure 14 is a schematic diagram showing an example of an image displayed on the display unit 1013 in the anomaly detection process according to the second embodiment. In the explanation of Figure 13, parts that are common with the flowchart in Figure 9 described above will be omitted as appropriate.

[0111] In Figure 13, in step S20, the sensing system 1b scans the entire area of ​​the scanning range 40 using the optical detection distance measuring unit 11 of the sensor unit 10 and acquires a point cloud within the scanning range 40. In the next step S21, the sensing system 1b generates a 2D image of the scanning range 40 based on the point cloud acquired in the scan of step S20 using the vibration distribution generation unit 125, and generates display control information for displaying the 2D image in 2D display mode. Since the point cloud only contains brightness information for display purposes, the 2D image displayed in this 2D display mode is a monochrome image. This 2D image is displayed, for example, on the display unit 1013 of the information processing device 1000.

[0112] In the next step, S22, the sensing system 1b determines whether or not an ROI has been set for the point cloud acquired in step S20 within the scanning range 40. The ROI may be set according to user operation, for example, as explained in step S12 of the flowchart in Figure 9. If the vibration distribution generation unit 125 determines that no ROI has been set (step S22, "No"), the sensing system 1b returns to step S22. On the other hand, if the vibration distribution generation unit 125 determines that an ROI has been set (step S22, "Yes"), the sensing system 1b proceeds to step S23.

[0113] Image 400a, shown in the upper left of Figure 14, shows an example of a 2D image in which an ROI 401 is defined by a rectangle. Image 400a is a 2D image based on a point cloud acquired by scanning the entire scanning range 40 with the optical detection range measuring unit 11a. Therefore, image 400a has a resolution corresponding to the points 2201, 2202, 2203, ... from which the laser beam is emitted by the optical detection range measuring unit 11a within the scanning range 40.

[0114] In step S23, the sensing system 1b scans the ROI 401 with the optical detection distance measuring unit 11. More specifically, the sensing system 1b generates a local scanning control signal for scanning the ROI 401 with the local scanning control unit 170 and outputs it to the optical detection distance measuring unit 11a. The optical detection distance measuring unit 11a scans the ROI 401 according to the local scanning control signal supplied from the local scanning control unit 170.

[0115] In step S23, the optical detection rangefinder 11a can scan the ROI 401 at a higher density than the scanning of the entire area in step S20. For example, the local scanning control unit 170 generates a local scanning control signal that controls the optical detection rangefinder 11a to narrow the spacing between the chirp light emission points compared to when scanning the entire area, and supplies this signal to the optical detection rangefinder 11a. By performing high-density scanning in this way compared to scanning the entire area, it is possible to acquire a point cloud with higher resolution than that acquired by scanning the entire area.

[0116] In the next step, S24, the sensing system 1b uses the vibration distribution generation unit 125 to display an image of ROI 401 on the display unit 1013 in 3D display mode, based on the point cloud acquired by scanning the ROI in step S23. Because ROI 401 was scanned at high density in step S23, the image displayed here has higher resolution than the 2D image displayed in step S21. In 3D display mode, the vibration distribution generation unit 125 displays candidate targets for detecting vibration distributions included in ROI 401, based on the 3D recognition information, which is the object recognition result from the 3D object recognition unit 122.

[0117] Image 400b, shown in the upper right of Figure 14, shows an example of an enlarged view of ROI 401 in Image 400a in the upper left of the same figure, along with the display of target candidates. In the example of Image 400b, each region 410a to 410e, which was extracted as a target candidate based on 3D recognition information, is shown with diagonal lines. These regions 410a to 410e were recognized by the 3D object recognition unit 122, for example, as different parts within ROI 401.

[0118] In Figure 13, in the next step S25, the sensing system 1b determines whether a target for vibration distribution detection has been selected from each of the regions 410a to 410e presented as target candidates in step S24, using the vibration distribution generation unit 125.

[0119] For example, in step S24, the sensing system 1b selects a target for vibration distribution detection from each region 410a to 410e in response to user operation on the image 400b displayed by the display unit 1013. If the vibration distribution generation unit 125 determines that no target has been set (step S25, "No"), the sensing system 1b returns to step S25. On the other hand, if the vibration distribution generation unit 125 determines that a target has been selected (step S25, "Yes"), the sensing system 1b proceeds to step S26.

[0120] Image 400c, shown in the lower left of Figure 14, illustrates an example where region 410b is selected as the target from among the target candidates in step S25. It is also possible to select multiple regions as targets.

[0121] In the next step, S26, the sensing system 1b scans the target region (region 410b in this example) selected in step S25 using the optical detection distance measuring unit 11. More specifically, the sensing system 1b generates a local scanning control signal for scanning the target region using the local scanning control unit 170 and outputs it to the optical detection distance measuring unit 11a. The optical detection distance measuring unit 11a scans the target region according to the local scanning control signal supplied from the local scanning control unit 170.

[0122] In step S26, the optical detection distance measuring unit 11a can scan the target region at a higher density than the scanning of the entire region in step S20 or the scanning of ROI 401 in step S23. For example, the local scanning control unit 170 can generate a local scanning control signal that controls the optical detection distance measuring unit 11a to further narrow the spacing between the chirp light emission points compared to when scanning the entire ROI 401. By performing a high-density scan of the target region in this way compared to scanning the entire region or ROI 401, it is possible to acquire a point cloud with a higher resolution than the point cloud acquired by scanning the entire region or ROI 401.

[0123] In the next step, S27, the sensing system 1b uses the vibration distribution generation unit 125 to detect the vibration distribution for the target region scanned in step S26 (region 410b in this example) and outputs the vibration distribution for the target region. The vibration distribution generation process by the vibration distribution generation unit 125 will be described later.

[0124] Image 400d, shown in the lower right of Figure 14, is a schematic representation of the vibration distribution in the target region (region 410b in this example), generated by the vibration distribution generation unit 125. In image 400d, the target region 410b is divided into regions 420a to 420d according to the degree of vibration, and the vibration distribution in region 410b is shown.

[0125] In the next step S28, the sensing system 1b, using the anomaly detection unit 20, determines whether there is a region in the target area where the degree of vibration exceeds a threshold, based on the vibration distribution output from the vibration distribution generation unit 125. The vibration distribution generation unit 125 performs a threshold determination on the aforementioned degree of vibration, for example.

[0126] If the anomaly detection unit 20 determines that there is a region within the target region where the degree of vibration exceeds a threshold (step S28, "Yes"), the sensing system 1b proceeds to step S29 and determines that an anomaly has been detected in the target region. On the other hand, if the anomaly detection unit 20 determines that there is no region within the target region where the degree of vibration exceeds a threshold (step S28, "No"), the sensing system 1b proceeds to step S30 and determines that there is no anomaly in the target region.

[0127] After the processing in step S29 or step S30, the series of processes shown in the flowchart in Figure 13 is completed.

[0128] Figure 15 is a flowchart illustrating an example of the vibration distribution generation process according to the second embodiment. The flowchart in Figure 15 is an example that shows the processes of steps S26 and S27 in Figure 13 described above in more detail, and is executed, for example, in the vibration distribution generation unit 125. In the explanation of Figure 15, parts that are common with the flowchart in Figure 11 described above will be omitted as appropriate.

[0129] In Figure 15, the processes in steps S200 to S204 on the left side measure the vibration distribution in the depth direction and the field of view direction using velocity point clouds. The processes in steps S210 to S214 on the right side measure the vibration distribution in the field of view direction using luminance point clouds.

[0130] First, we will explain the vibration distribution measurement process in the depth direction and field of view direction using velocity point clouds, from step S200 to step S204.

[0131] The process in step S200 in Figure 15 corresponds to the process in step S26 in the flowchart of Figure 13. In step S200, the vibration distribution generation unit 125 acquires a point cloud frame of velocity point clouds in the target region based on the output of the optical detection distance measurement unit 11. In the next step S201, the vibration distribution generation unit 125 acquires 3D recognition information obtained by the recognition process performed by the 3D object recognition unit 122 on this point cloud frame of velocity point clouds. In the next step S202, the vibration distribution generation unit 125 extracts velocity point clouds from the point cloud frame. The velocity point clouds extracted in step S02 have 3D information. The vibration distribution generation unit 125 may store the extracted velocity point clouds of the target region in the storage unit 126a.

[0132] In the next step, S203, the vibration distribution generation unit 125 determines whether it has performed measurements on a predetermined number of point cloud frames necessary for detecting vibrations in the depth direction. If the vibration distribution generation unit 125 determines that it has not performed measurements on a predetermined number of point cloud frames (step S203, "No"), it returns to step S200 to acquire the next point cloud frame and perform measurements on the acquired point cloud frame (steps S201, S202).

[0133] On the other hand, if the vibration distribution generation unit 125 determines that it has performed measurements of a predetermined number of point cloud frames (step S203, "Yes"), it proceeds to step S204.

[0134] In step S204, the vibration distribution generation unit 125 calculates the vibration distribution in the depth direction and the vibration distribution in the field of view direction based on the velocity point cloud obtained in the processing up to step S203, for example, using the method described with reference to Figures 2 and 3.

[0135] Next, we will explain the vibration distribution measurement process in the field of view direction using the luminance point cloud, which takes place in steps S210 to S214.

[0136] The process in step S210 in Figure 15 corresponds to the process in step S26 in the flowchart of Figure 13. In step S210, the vibration distribution generation unit 125 acquires a point cloud frame of luminance points in the target region based on the output of the optical detection distance measurement unit 11. In the next step S211, the vibration distribution generation unit 125 acquires 3D recognition information obtained by the recognition process performed by the 3D object recognition unit 122 on this point cloud frame of luminance points.

[0137] In the next step S212, the vibration distribution generation unit 125 extracts luminance point clouds from the point cloud frame. The vibration distribution generation unit 125 extracts 2D information from the point cloud frame made up of luminance point clouds. For example, the vibration distribution generation unit 125 projects the information of each point included in the point cloud frame onto a plane in the field of view direction. Therefore, the luminance point cloud extracted in step S212 has 2D information. The vibration distribution generation unit 125 may store the extracted 2D information of the target region in the storage unit 126a.

[0138] In the next step, S213, the vibration distribution generation unit 125 determines whether or not it has performed measurements on a predetermined number of point cloud frames necessary for detecting vibrations in the field of view direction. If the vibration distribution generation unit 125 determines that it has not performed measurements on a predetermined number of point cloud frames (step S213, "No"), it returns to step S200 to acquire the next point cloud frame and perform measurements on the acquired point cloud frame (steps S211, S212).

[0139] On the other hand, if the vibration distribution generation unit 125 determines that it has performed measurements of a predetermined number of point cloud frames (step S213, "Yes"), it proceeds to step S214.

[0140] In step S214, the vibration distribution generation unit 125 calculates the vibration distribution in the field of view direction based on the luminance point clouds for multiple frames, each of which is 2D information, acquired in the processing up to step S213, for example, using the method described using section (c) of Figure 1.

[0141] After completing the processing in steps S204 and S214, the vibration distribution generation unit 125 moves the processing to step S220. In step S220, the vibration distribution generation unit 125 integrates the vibration distributions in the depth direction and field of view direction calculated in step S204 with the vibration distribution in the field of view direction calculated in step S214 to obtain the vibration distribution of the target area, and outputs vibration distribution information showing the obtained vibration distribution to the anomaly detection unit 20.

[0142] Similar to the flowchart in Figure 11, in the flowchart in Figure 15, the processes in steps S200 to S204 and steps S210 to S214 can be executed independently of each other. However, these processes in steps S200 to S204 and steps S210 to S214 may be executed synchronously. Furthermore, these processes in steps S200 to S204 and steps S210 to S214 may be processes performed by the same scan over a common scan range. In this case, the predetermined numbers determined in steps S203 and S213 will be the same numbers in steps S201 and S213.

[0143] Thus, in the second embodiment, 3D object recognition processing is performed on the point cloud output by the optical detection ranging unit 11a, which measures distance using FMCW-LiDAR, and the point cloud of the target region is extracted based on the recognition result. In this second embodiment, a narrow range is defined as the target region relative to the scanning range 40 of the entire area of ​​the optical detection ranging unit 11a, and operations on the target region are performed at a higher density relative to the scanning of the scanning range 40. As a result, it is possible to measure the vibration distribution within the target region with higher accuracy, and anomaly detection of the object based on the measured vibration distribution can be performed with higher accuracy.

[0144] (5. Third Embodiment) Next, a third embodiment of the present disclosure will be described. The third embodiment is an example in which, in addition to the optical detection distance measuring unit 11a, an imaging device is provided in the sensor unit 10a according to the second embodiment described above, and object recognition is performed using the point cloud acquired by the optical detection distance measuring unit 11a and the captured image taken by the imaging device to obtain recognition information.

[0145] An imaging device capable of acquiring captured images containing information for each color (R, G, and B) generally has a much higher resolution than the optical detection rangefinder 11a of an FMCW-LiDAR. Therefore, by performing recognition processing using both the optical detection rangefinder 11a and the imaging device, it becomes possible to perform detection and recognition processing with higher accuracy compared to performing detection and recognition processing using only point cloud information from the optical detection rangefinder 11a.

[0146] Figure 16 is a block diagram showing the configuration of an example of a sensing system according to the third embodiment. Note that detailed explanations of parts common to Figure 12 described above will be omitted below.

[0147] In Figure 16, the sensing system 1c according to the third embodiment includes a sensor unit 10b, a signal processing unit 12c, and an anomaly detection unit 20a.

[0148] The sensor unit 10b includes a light detection distance measuring unit 11a and a camera 13. The camera 13 is an imaging device that includes an image sensor capable of acquiring captured images having the above-mentioned RGB color information (hereinafter referred to as color information as appropriate), and is capable of controlling the field of view and the imaging range at the entire field of view in accordance with a field of view control signal supplied from an external source.

[0149] The image sensor includes, for example, a pixel array in which pixels that each output a signal corresponding to the light they receive are arranged in a two-dimensional grid, and a drive circuit for driving each pixel in the pixel array. The camera 13 also includes, for example, a zoom mechanism and an imaging direction control mechanism, and is capable of changing the field of view and imaging direction according to the field of view control signal, thereby magnifying and imaging a desired subject within predetermined limits. These zoom mechanism and imaging direction control mechanism may be optical or electronic.

[0150] In Figure 16, the signal processing unit 12c includes a point cloud synthesis unit 160, a 3D object detection unit 121a, a 3D object recognition unit 122a, an image synthesis unit 150, a 2D (Two Dimensions) object detection unit 151, a 2D object recognition unit 152, and an I / F unit 123a.

[0151] The point cloud synthesis unit 160, the 3D object detection unit 121a, and the 3D object recognition unit 122a perform processing related to point cloud information. In addition, the image synthesis unit 150, the 2D object detection unit 151, and the 2D object recognition unit 152 perform processing related to the captured image.

[0152] The point cloud merging unit 160 acquires a point cloud from the optical detection distance measuring unit 11a and an image captured from the camera 13. Based on the point cloud and the image captured, the point cloud merging unit 160 combines color information and other information to generate a composite point cloud, which is a point cloud in which new information is added to each measurement point of the point cloud.

[0153] More specifically, the point cloud compositing unit 160 references the pixels of the captured image corresponding to the angular coordinates of each measurement point in the point cloud through coordinate system transformation, and acquires representative color information for each measurement point. The measurement points correspond to the points where reflected light was received for each point 2201, 2202, 2203, ... as explained using Figure 6. The point cloud compositing unit 160 adds the acquired color information for each measurement point to the measurement information for each measurement point. The point cloud compositing unit 160 outputs a composite point cloud in which each measurement point has 3D coordinate information, velocity information, luminance information, and color information.

[0154] Furthermore, it is preferable that the coordinate system transformation between the point cloud and the captured image is performed after, for example, a calibration process based on the positional relationship between the optical detection rangefinder 11a and the camera 13 has been performed in advance, and this calibration result has been reflected in the angular coordinates of the velocity point cloud and the pixel coordinates in the captured image.

[0155] The 3D object detection unit 121a acquires the composite point cloud output from the point cloud synthesis unit 160 and detects measurement points that indicate 3D objects included in the acquired composite point cloud. The 3D object detection unit 121a extracts the point cloud of measurement points that indicate 3D objects detected from the composite point cloud as a localized point cloud. The extraction process of the localized point cloud and the generation process of region information by the 3D object detection unit 121a are equivalent to the processes in the 3D object detection unit 121 described with reference to Figure 8, so a detailed explanation is omitted here.

[0156] The 3D object detection unit 121a outputs a group of localized points, 3D coordinates relating to the group of localized points, velocity information, and brightness information as 3D detection information indicating the 3D detection result. The 3D detection information is supplied to the 3D object recognition unit 122a and the 2D object detection unit 151, which will be described later. At this time, the 3D object detection unit 121a may add label information indicating the 3D object corresponding to the detected group of localized points to the region of the group of localized points, and include the added label information in the 3D detection result.

[0157] The 3D object recognition unit 122a acquires 3D detection information output from the 3D object detection unit 121a. The 3D object recognition unit 122a also acquires region information and attribute information output from the 2D object recognition unit 152, which will be described later. Based on the acquired 3D detection information and the region information acquired from the 2D object recognition unit 152, the 3D object recognition unit 122a performs object recognition on the localized point cloud.

[0158] The 3D object recognition unit 122a performs point cloud recognition processing on the localized point cloud if the number of points included in the localized point cloud is greater than or equal to a predetermined number that can be used to recognize the target object, based on the 3D detection information and region information. The 3D object recognition unit 122a estimates attribute information about the recognized object through this point cloud recognition processing. Hereinafter, attribute information based on the point cloud will be referred to as 3D attribute information.

[0159] The 3D object recognition unit 122a may perform object recognition processing on the localized point cloud corresponding to the 3D object among the point cloud output from the optical detection distance measurement unit 11a. For example, the 3D object recognition unit 122a can remove the point cloud portion other than the localized point cloud from the point cloud output from the optical detection distance measurement unit 11a and not perform object recognition processing on that portion. This makes it possible to reduce the processing load on the recognition unit 122a.

[0160] The 3D object recognition unit 122a, when the confidence level of the estimated 3D attribute information is above a certain level, that is, when the recognition process can be successfully executed, integrates time information indicating the time of measurement, 3D region information, and 3D attribute information and outputs it as 3D recognition information.

[0161] Attribute information, as a result of recognition processing, is information that indicates the attributes of the target object, such as the type of object to which the unit belongs or its unique classification, for each point in a point cloud or pixel in an image. If the target object is a person, 3D attribute information can be expressed as a unique numerical value belonging to that person, for example, assigned to each point in the point cloud.

[0162] The image synthesis unit 150 acquires a velocity point cloud from the optical detection distance measurement unit 11a and acquires an image from the camera 13. Based on the velocity point cloud and the image, the image synthesis unit 150 generates a distance image and a velocity image. The distance image is an image that includes information indicating the distance from the measurement point. The velocity image is an image due to the Doppler effect and includes information such as the velocity relative to the measurement point and the direction of the velocity.

[0163] The image synthesis unit 150 synthesizes the distance image and velocity image with the captured image, matching their coordinates through coordinate transformation, to generate a composite image using RGB. The composite image generated here is an image in which each pixel contains information about color, distance, and velocity. Note that the distance image and velocity image have a lower resolution than the captured image output from the camera 13. Therefore, the image synthesis unit 150 may perform processing such as upscaling on the distance image and velocity image to match the resolution of the captured image.

[0164] The image synthesis unit 150 outputs the generated composite image. The composite image refers to an image in which new information is added to each pixel of the image by combining distance, velocity, and other information. The composite image includes 2D coordinate information, color information, distance information, velocity information, and brightness information for each pixel. The composite image is supplied to the 2D object detection unit 151 and the I / F unit 123a.

[0165] The 2D object detection unit 151 extracts a partial image corresponding to the 3D region information output from the 3D object detection unit 121a from the composite image supplied by the image synthesis unit 150. The 2D object detection unit 151 also detects an object from the extracted partial image and generates region information that indicates, for example, the smallest rectangular area containing the detected object. This region information based on the captured image is called 2D region information. The 2D region information is represented as a set of points and pixels where the values ​​assigned to each measurement point or pixel by the optical detection distance measurement unit 11a fall within a specified range.

[0166] The 2D object detection unit 151 outputs the generated partial image and 2D region information as 2D detection information.

[0167] The 2D object recognition unit 152 acquires a partial image included in the 2D detection information output from the 2D object detection unit 151, and performs image recognition processing such as inference on the acquired partial image to estimate attribute information related to the partial image. In this case, the attribute information is expressed as a unique numerical value indicating that it belongs to a vehicle, for example, if the object is a vehicle, assigned to each pixel of the image. Hereinafter, attribute information based on a partial image (captured image) will be referred to as 2D attribute information.

[0168] The 2D object recognition unit 152 outputs 2D recognition information by integrating the 2D coordinate information, velocity information, attribute information, and reliability for each pixel, along with the 2D size information, when the confidence level of the estimated 2D attribute information is above a certain level, i.e., the recognition process can be performed successfully. If the confidence level of the estimated 2D attribute information is below a certain level, the 2D object recognition unit 152 may output the information excluding the attribute information by integrating each piece of information.

[0169] The I / F unit 123a receives the composite point cloud output from the point cloud synthesis unit 160 and the 3D recognition information output from the 3D object recognition unit 122a as inputs. The I / F unit 123a also receives the composite image output from the image synthesis unit 150 and the 2D recognition information output from the 2D object recognition unit 152 as inputs. The I / F unit 123a selects the information to output from the input composite point cloud, 3D recognition information, composite image, and 2D recognition information, for example, according to external settings.

[0170] The local scanning control unit 170, similar to the local scanning control unit 170 in Figure 12, sets a target area, which is a narrower scanning range than the entire scanning range 40 of the optical detection distance measuring unit 11a, within the scanning range 40, based on the target area setting information and the 3D recognition information output from the 3D object recognition unit 122a. The local scanning control unit 170 outputs a local scanning control signal for scanning the set target area.

[0171] The field of view control unit 171 sets a target area that is a narrow field of view relative to the entire field of view (entire imaging range) of the camera 13, based on the target area setting information and the 2D recognition information output from the 2D object recognition unit. Here, common target area setting information is input to both the field of view control unit 171 and the local scanning control unit 170. Therefore, the target area set by the field of view control unit 171 is set to have the same position and size as the target area set by the local scanning control unit 170.

[0172] The I / F unit 123a receives the composite point cloud output from the point cloud synthesis unit 160 and the 3D recognition information output from the 3D object recognition unit 122a. In addition, the I / F unit 123a receives the composite image output from the image synthesis unit 150 and the 2D recognition information output from the 2D object recognition unit 152.

[0173] The I / F unit 123a outputs the composite point cloud of the entire region supplied from the point cloud synthesis unit 160 and the composite image of the entire region supplied from the image synthesis unit 150 to the anomaly detection unit 20a. The I / F unit 123a also outputs the composite point cloud of the entire region, the composite image of the entire region, 3D recognition information supplied from the 3D object recognition unit 122a and 2D recognition information supplied from the 2D object recognition unit 152 to the vibration distribution generation unit 125a.

[0174] The vibration distribution generation unit 125a estimates the vibration distribution in the object 50 and generates vibration distribution information based on the composite point cloud of the entire region, the composite image of the entire region, 3D recognition information, and 2D recognition information supplied from the I / F unit 123a. The vibration distribution generation unit 125a may estimate the vibration distribution of the object 50 using the supplied information (composite point cloud of the entire region, composite image of the entire region, 3D recognition information, and 2D recognition information) and past information of the same information stored in the storage unit 126a.

[0175] The vibration distribution generation unit 125a supplies vibration distribution information, which represents the estimated vibration distribution, to the anomaly detection unit 20a. The vibration distribution generation unit 125a also stores the composite point cloud of the entire region, the composite image of the entire region, 3D recognition information, and 2D recognition information in the storage unit 126a as historical information.

[0176] Furthermore, the vibration distribution generation unit 125a can generate display control information for displaying an image to be presented to the user based on the composite point cloud of the entire region, the composite image of the entire region, 3D recognition information, and 2D recognition information supplied from the I / F unit 123a.

[0177] The anomaly detection unit 20a detects anomalies in the object 50 based on the composite point cloud and composite image of the entire area supplied by the signal processing unit 12c, and vibration distribution information. For example, the anomaly detection unit 20a may generate an evaluation value based on the vibration distribution information and determine whether or not there is an anomaly in the object 50 by performing a threshold judgment on the generated evaluation value. The anomaly detection unit 20a outputs the anomaly detection result for the object 50 to an external source, for example.

[0178] Figure 17 is a flowchart illustrating an example of the anomaly detection process according to the third embodiment. Figure 18 is a schematic diagram showing an example of an image displayed on the display unit 1013 in the anomaly detection process according to the third embodiment. In the explanation of Figure 17, parts that are common with the flowchart in Figure 13 described above will be omitted as appropriate.

[0179] In Figure 17, in step S40, the sensing system 1c scans the entire area of ​​the scanning range 40 using the optical detection distance measuring unit 11a of the sensor unit 10b and acquires a point cloud within the scanning range 40. Also in step S40, the sensing system 1c uses the camera 13 to capture an image in the imaging range corresponding to the scanning range 40 and acquires the captured image.

[0180] In the next step S41, the sensing system 1c generates a 2D image relating to the scanning range 40 based on the image captured by the camera 13 in step S40 using the vibration distribution generation unit 125a, and generates display control information for displaying the 2D image in 2D display mode. Since the image captured by the camera 13 has color information for each RGB color, the 2D image displayed in this 2D display mode is a color image. In general, the image captured by the camera 13 has a much higher resolution than the point cloud acquired by the light detection distance measurement unit 11a, and the 2D image is also a high-resolution image. The 2D image is displayed, for example, on the display unit 1013 of the information processing device 1000.

[0181] In the next step, S42, the sensing system 1c determines whether or not an ROI has been set for the scanning range 40 acquired in step S40. The ROI may be set according to user operation, for example, as explained in step S22 of the flowchart in Figure 13. If the vibration distribution generation unit 125a determines that no ROI has been set (step S42, "No"), the sensing system 1c returns to step S42. On the other hand, if the vibration distribution generation unit 125a determines that an ROI has been set (step S42, "Yes"), the sensing system 1c proceeds to step S43.

[0182] Image 500a, shown in the upper left of Figure 18, is an example of a 2D image in which an ROI 501 is defined by a rectangle. Image 500a is a 2D image based on an image captured by camera 13. Therefore, image 500a has a resolution corresponding to the pixel arrangement in camera 13.

[0183] In step S43, the sensing system 1c scans the ROI 501 with the optical detection ranging unit 11a. More specifically, the sensing system 1c generates a local scanning control signal for scanning the ROI 501 with the local scanning control unit 170 and outputs it to the optical detection ranging unit 11a. The optical detection ranging unit 11a scans the ROI 501 according to the local scanning control signal supplied from the local scanning control unit 170. As explained in step S23 of Figure 13, in step S43, the optical detection ranging unit 11a can scan the ROI 501 at a higher density than the scanning of the entire area in step S40, and can acquire a point cloud with higher resolution.

[0184] In the next step, S44, the sensing system 1c displays an image of ROI 501 on the display unit 1013 in 3D display mode, based on the point cloud acquired by scanning the ROI in step S43, using the vibration distribution generation unit 125a. The image displayed here is an image based on a composite point cloud, which is created by combining the captured image and the point cloud using the point cloud synthesis unit 160. Therefore, the image of ROI 501 displayed in step S44 is a higher-resolution image compared to, for example, the image of ROI 401 displayed in step S24 in Figure 13. In 3D display mode, the vibration distribution generation unit 125a displays candidate targets for detecting vibration distributions included in ROI 501, based on the 3D recognition information, which is the object recognition result by the 3D object recognition unit 122a.

[0185] Image 500b, shown in the upper right of Figure 18, shows an example of an enlarged view of ROI 501 in Image 500a in the upper left of the same figure, along with the display of target candidates. In the example of Image 500b, each region 510a to 510e, which was extracted as a target candidate based on 3D recognition information, is shown with diagonal lines. These regions 510a to 510e were recognized by the 3D object recognition unit 122a, for example, as different parts within ROI 501.

[0186] In Figure 17, in the next step S45, the sensing system 1c determines whether a target for vibration distribution detection has been selected from each of the regions 510a to 510e presented as target candidates in step S44, using the vibration distribution generation unit 125a.

[0187] For example, in step S44, the sensing system 1c selects a target for vibration distribution detection from each region 510a to 510e in response to user operation on the image 500b displayed by the display unit 1013. If the vibration distribution generation unit 125a determines that no target has been set (step S45, "No"), the sensing system 1c returns to step S45. On the other hand, if the vibration distribution generation unit 125a determines that a target has been selected (step S45, "Yes"), the sensing system 1c proceeds to step S46.

[0188] Image 500c, shown in the lower left of Figure 18, illustrates an example where region 510b is selected as the target from among the target candidates in step S45. It is also possible to select multiple regions as targets.

[0189] In the next step, S46, the sensing system 1c scans the target region (region 510b in this example) selected in step S45 using the optical detection distance measuring unit 11a.

[0190] More specifically, the sensing system 1c generates a local scanning control signal for scanning the target area using the local scanning control unit 170 and outputs it to the optical detection distance measuring unit 11a. The optical detection distance measuring unit 11a scans the target area according to the local scanning control signal supplied from the local scanning control unit 170. The sensing system 1c also generates a field of view control signal for setting the target area as the imaging range using the field of view control unit 171 and outputs it to the camera 13. The field of view control signal includes, for example, zoom control information for changing the field of view used for imaging to the field of view corresponding to the target area, and imaging direction control information for imaging the direction of the target area.

[0191] Similar to the explanation in step S26 of Figure 13, in step S46, the optical detection distance measuring unit 11a can scan the target region at a higher density than the scanning of the entire region in step S40 or the scanning of ROI 501 in step S43. Therefore, it is possible to acquire a point cloud with a higher resolution than the point cloud acquired by scanning the entire region or scanning ROI 501.

[0192] Furthermore, camera 13 can image the target area at its own resolution by, for example, performing an optical zoom operation. Also, if the resolution of camera 13 is sufficiently high, even if the zoom operation is performed electronically (for example, by magnification through image processing), it is possible to image the target area at a higher resolution compared to a point cloud.

[0193] In the next step, S47, the sensing system 1c uses the vibration distribution generation unit 125a to detect the vibration distribution in the target region (region 510b in this example) scanned and imaged in step S46, and outputs the vibration distribution in the target region. The vibration distribution generation process by the vibration distribution generation unit 125a will be described later.

[0194] Image 500d, shown in the lower right of Figure 18, is a schematic representation of the vibration distribution in the target region (region 510b in this example), generated by the vibration distribution generation unit 125a. In image 500d, the target region 510b is divided into regions 520a to 520d according to the degree of vibration, and the vibration distribution in region 510b is shown.

[0195] In the next step S48, the sensing system 1c, using the anomaly detection unit 20a, determines whether there is a region in the target area where the degree of vibration exceeds a threshold, based on the vibration distribution output from the vibration distribution generation unit 125a. The vibration distribution generation unit 125a performs a threshold determination on the aforementioned degree of vibration, for example.

[0196] If the sensing system 1c determines, based on the anomaly detection unit 20a, that there is a region within the target region where the degree of vibration exceeds a threshold (step S48, "Yes"), it proceeds to step S49 and determines that an anomaly has been detected in the target region. On the other hand, if the sensing system 1c determines, based on the anomaly detection unit 20a, that there is no region within the target region where the degree of vibration exceeds a threshold (step S48, "No"), it proceeds to step S50 and determines that there is no anomaly in the target region.

[0197] After the processing in step S49 or step S50, the series of processes shown in the flowchart in Figure 17 is completed.

[0198] Figure 19 is a flowchart illustrating an example of the vibration distribution generation process according to the third embodiment. The flowchart in Figure 19 is an example that shows the processes of steps S46 and S47 in Figure 17 described above in more detail, and is executed, for example, in the vibration distribution generation unit 125a. In the explanation of Figure 19, parts that are common with the flowchart in Figure 15 described above will be omitted as appropriate.

[0199] In Figure 19, the processes in steps S200 to S204 on the left measure the vibration distribution in the depth direction and the field of view direction using velocity point clouds. The processes in steps S210 to S214 in the center and steps S410 to S414 on the left measure the vibration distribution in the field of view direction using luminance information, respectively. Here, the processes in steps S210 to S214 are based on the luminance point cloud acquired by the light detection distance measuring unit 11a, and the processes in steps S410 to S414 are based on the captured image acquired by the camera 13.

[0200] Note that the processes in steps S200 to S204 and steps S210 to S214 are the same as those in steps S200 to S214 and steps S210 to S214 explained in Figure 15, so their explanation is omitted here. Here, the processes in steps S200 and S210, and step S410 in Figure 19 correspond to the process in step S46 in the flowchart of Figure 17, respectively.

[0201] When the processing according to the flowchart in Figure 19 begins, the vibration distribution generation unit 125a executes the processes from steps S200 to S204. At the same time, in step S400, the vibration distribution generation unit 125a branches the processing into the processes from steps S210 to S214 and the processes from steps S410 to S414, depending on the resolution of the point cloud acquired by the optical detection distance measurement unit 11a and the resolution of the captured image acquired by the camera 13.

[0202] In other words, generally the camera 13 has a higher resolution than the optical distance measuring unit 11a, but depending on the target area set in step S45 of the flowchart in Figure 17, the resolution of the optical distance measuring unit 11a may be higher than the resolution of the camera 13. For example, if the target area is narrowed to a very small range, the resolution of the point cloud acquired by the optical distance measuring unit 11a may be higher than the resolution of the image captured by the camera 13. Also, the relationship between the resolution of the optical distance measuring unit 11a and the resolution of the camera 13 may change depending on the specifications of the optical distance measuring unit 11a and the camera 13.

[0203] If the resolution of the point cloud is higher than the resolution of the captured image (step S400, "Yes"), the vibration distribution generation unit 125a proceeds to step S210. On the other hand, if the resolution of the point cloud is less than or equal to the resolution of the captured image (step S400, "No"), the vibration distribution generation unit 125a proceeds to step S410.

[0204] In step S410, the vibration distribution generation unit 125a acquires an image frame from the image of the target region output from the camera 13. In the next step S411, the vibration distribution generation unit 125a acquires 2D recognition information obtained by the recognition processing of the image frame of the target region by the 2D object recognition unit 152. In the next step S412, the vibration distribution generation unit 125a extracts an image of the target region from the image acquired in step S410. The vibration distribution generation unit 125a may store the extracted image of the target region in the storage unit 126a.

[0205] In the next step, S413, the vibration distribution generation unit 125a determines whether or not it has performed measurements on a predetermined number of image frames necessary for detecting vibrations in the field of view direction. If the vibration distribution generation unit 125a determines that it has not performed measurements on a predetermined number of image frames (step S413, "No"), it returns to step S400 to acquire the next image frame and perform measurements on the acquired image frame (steps S411, S412).

[0206] On the other hand, if the vibration distribution generation unit 125a determines that it has performed measurements on a predetermined number of image frames (step S413, "Yes"), it proceeds to step S414.

[0207] In step S414, the vibration distribution generation unit 125a calculates the vibration distribution in the field of view direction based on the image frames of multiple frames acquired in the processing up to step S413, for example, using the method described using section (c) of Figure 1.

[0208] After the processing in steps S204 and S214 is completed, or after the processing in steps S204 and S414 is completed, the vibration distribution generation unit 125a moves the processing to step S420. In step S420, the vibration distribution generation unit 125a integrates the vibration distribution in the depth direction and field of view direction calculated in step S204 with the vibration distribution in the field of view direction calculated in step S214 or step S414 to obtain the vibration distribution of the target area, and outputs vibration distribution information showing the obtained vibration distribution to the anomaly detection unit 20a.

[0209] Similar to the flowchart in Figure 11, in the flowchart in Figure 19, the processes of steps S200 to S204 and the processes of steps S210 to S214 or steps S410 to S414 can be executed independently of each other. However, these processes of steps S200 to S204 and steps S210 to S214 or steps S410 to S414 may be executed synchronously. Furthermore, steps S200 to S204 and steps S210 to S214 may be processes performed by the same scan over a common scan range. In this case, the predetermined numbers determined in steps S203 and S213 will be the same numbers as in steps S201 and S213.

[0210] Thus, in the third embodiment, 3D object recognition processing is performed on the point cloud output by the optical detection ranging unit 11a, which measures distance using FMCW-LiDAR, and the point cloud of the target region is extracted based on the recognition result. In this third embodiment, a narrow range is defined as the target region relative to the scanning range 40 of the entire area of ​​the optical detection ranging unit 11a, and operations on the target region are performed at a high density relative to the scanning of the scanning range 40.

[0211] In the third embodiment, since the camera 13 is used to acquire an image containing color information for each RGB color, it becomes possible to colorize the image based on the point cloud output by the optical detection range measuring unit 11a, making it easier for the user to select a target area. In addition, generally, the image output from the camera 13 has a higher resolution than the point cloud output from the optical detection range measuring unit 11a, so it is possible to measure the vibration distribution within the target area with higher accuracy, and anomaly detection of the object based on the measured vibration distribution can be performed with higher accuracy.

[0212] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0213] Furthermore, this technology can also be configured as follows. (1) A first recognition unit includes an optical transmission unit that transmits light modulated by a frequency continuous modulation wave, and an optical reception unit that receives light and outputs a received signal, and performs recognition processing based on the point cloud output by an optical detection distance measuring unit that outputs a point cloud containing a plurality of points each having velocity information based on the received signal, and outputs 3D recognition information of the object, A generation unit that generates vibration distribution information showing the vibration distribution of the object based on the velocity information and the three-dimensional recognition information, A detection unit that detects abnormalities in the object based on the vibration distribution information, Equipped with, Information processing device. (2) The system further includes a scanning control unit that generates a scanning control signal to control the scanning range of light transmitted by the optical transmitting unit, The detection unit is Based on the vibration distribution information generated by the generation unit using the point cloud acquired from the scanning range controlled by the scanning control signal, abnormality detection of the target object is performed. The information processing device described in (1) above. (3) The generating unit is Based on the velocity information of the point cloud and the three-dimensional recognition information, the system extracts three-dimensional information from the point cloud corresponding to the object, detects vibrations in the optical axis direction of the light transmitted by the optical transmitting unit of the object, and generates first vibration distribution information included in the vibration distribution information, which shows the distribution of the vibrations in the optical axis direction on a plane intersecting the optical axis direction. The information processing device described in (1) or (2) above. (4) The point cloud further includes brightness information indicating the brightness of each of the plurality of points included in the point cloud, The generating unit is Based on the luminance information and the three-dimensional recognition information, the point cloud corresponding to the object is extracted as two-dimensional information from the point cloud, and vibrations in the field of view direction of the optical detection distance measuring unit are detected, and a second vibration distribution information is generated, which is included in the vibration distribution information and shows the distribution of vibrations in the field of view direction. The detection unit is Based on the first vibration distribution information and the second vibration distribution information generated by the generation unit, abnormality detection of the target object is performed. The information processing device described in (3) above. (5) The generating unit is Based on the velocity information and 3D recognition information of the point cloud, a third vibration distribution information is generated, which is included in the vibration distribution information and indicates the distribution of vibrations in the field of view of the optical detection distance measuring unit, based on the point cloud corresponding to the object, which is extracted as 3D information from the point cloud. The detection unit is Based on the first vibration distribution information, the second vibration distribution information, and the third vibration distribution information generated by the generation unit, abnormality detection of the target object is performed. The information processing device described in (4) above. (6) The first recognition unit described above is: Based on the point cloud output by the optical detection distance measuring unit, the recognition process is performed on the point cloud that is included in the target region set for that point cloud. An information processing device as described in any of (1) to (5) above. (7) The aforementioned target area is, Of the point cloud output by the optical detection distance measuring unit, a region of interest is set based on the point cloud, The information processing device described in (6) above. (8) The system further includes a second recognition unit that performs recognition processing based on the captured image output by an image sensor that outputs a captured image containing color information based on the received light, and outputs two-dimensional recognition information of the object. The generating unit is Based on the velocity information, the captured image, the three-dimensional recognition information, and the two-dimensional recognition information, the vibration distribution information of the object is generated. An information processing device as described in any of (1) to (7) above. (9) The generating unit is Based on the velocity information of the point cloud and the three-dimensional recognition information, the point cloud corresponding to the object is extracted as three-dimensional information from the point cloud, and vibrations in the optical axis direction of the light transmitted by the optical transmitting unit of the object are detected, and the first vibration distribution information included in the vibration distribution information shows the distribution of the vibrations in the optical axis direction on a plane intersecting the optical axis direction; Based on the luminance information indicating the brightness of each of the multiple points included in the point cloud and the three-dimensional recognition information, a second vibration distribution information is included in the vibration distribution information, which is extracted as two-dimensional information from the point cloud corresponding to the object, based on the point cloud, and based on the vibration in the field of view direction of the optical detection distance measuring unit, the vibration in the field of view direction is detected, and the distribution of vibration in the field of view direction is shown; Based on the velocity information of the point cloud and the three-dimensional recognition information, a third vibration distribution information is included in the vibration distribution information, which is extracted as three-dimensional information from the point cloud corresponding to the object, and vibrations in the field of view direction of the optical detection distance measuring unit are detected based on the point cloud corresponding to the object, and the distribution of vibrations in the field of view direction is shown; Based on the two-dimensional recognition information output by the second recognition unit, a fourth vibration distribution information indicating the vibration distribution in the field of view direction is generated; Generate, The detection unit is Anomaly detection of the object is performed using the first vibration distribution information, the second vibration distribution information, the third vibration distribution information, and the fourth vibration distribution information. The information processing device described in (8) above. (10) The generating unit is Based on the resolution of the point cloud output from the optical detection distance measuring unit and the resolution of the image output from the image sensor, the system selects which of the second vibration distribution information and the fourth vibration distribution information to use as the vibration distribution information in the field of view direction. The information processing device described in (9) above. (11) The generating unit is When the resolution of the point cloud output from the optical distance measuring unit is higher than the resolution of the image output from the image sensor, the second vibration distribution information is selected. The fourth vibration distribution information is selected when the resolution of the image output from the image sensor is higher than the resolution of the point cloud output from the optical detection distance measuring unit. The information processing device described in (10) above. (12) The first recognition unit and the second recognition unit are Based on the target region set for the captured image output by the image sensor, the recognition process is performed accordingly. The information processing device described in any of (8) to (11) above. (13) The aforementioned target area is, Based on the region of interest set for the aforementioned captured image, The information processing device described in (12) above. (14) A first recognition step includes an optical transmission unit that transmits light modulated by a frequency continuous modulation wave, an optical reception unit that receives light and outputs a received signal, and an optical detection distance measuring unit that outputs a point cloud containing a plurality of points each having velocity information based on the received signal, and performs recognition processing based on the point cloud output and outputs 3D recognition information of the object, A generation step of generating vibration distribution information showing the vibration distribution of the object based on the velocity information and the three-dimensional recognition information, A detection step that detects an abnormality in the object based on the vibration distribution information, Having, Information processing methods. (15) An optical transmission unit that transmits light modulated by a frequency continuous modulation wave, and an optical reception unit that receives light and outputs a received signal, and an optical detection distance measuring unit that outputs a point cloud including a plurality of points each having velocity information based on the received signal, A first recognition unit performs recognition processing based on the point cloud output by the optical detection distance measuring unit and outputs three-dimensional recognition information of the object, A generation unit that generates vibration distribution information showing the vibration distribution of the object based on the velocity information and the three-dimensional recognition information, A detection unit that detects abnormalities in the object based on the vibration distribution information, Equipped with, Sensing system. [Explanation of symbols]

[0214] 1,1a,1b,1c Sensing System 10,10a Sensor Unit 11,11a Optical detection range measuring unit 12, 12a, 12b, 12c Signal Processing Unit 13 Cameras 20,20a Anomaly detection unit 40 Scanning range 41 scan lines 50 Objects 100 Scanning Unit 101 Optical Transmitter 103 Optical receiving unit 111 Scanning Control Unit 112 Angle detection unit 116 Transmitting Optical Control Unit 117 Received signal processing unit 130 Point cloud generator 121,121a 3D object detection unit 122,122a 3D object recognition unit 123,123a Interface section 125,125a Vibration distribution generation part 126,126a Storage section 151 2D Object Detection Unit 152 2D object recognition section 170 Local Scanning Control Unit 171 Angle of View Control Unit Images 300a, 300b, 300c, 300d, 400a, 400b, 400c, 400d, 500a, 500b, 500c, 500d 301, 401, 501 ROI 310a,310b,310c,310d,310e,320a,320b,320c,320d,410a,410b,410c,410d,41 0e,420a,420b,420c,420d,510a,510b,510c,510d,510e,520a,520b,520c,520d area

Claims

1. A first recognition unit includes an optical transmission unit that transmits light modulated by a frequency continuous modulation wave, and an optical reception unit that receives light and outputs a received signal, and performs recognition processing based on the point cloud output by an optical detection distance measuring unit that outputs a point cloud containing a plurality of points each having velocity information based on the received signal, and outputs three-dimensional recognition information of the object, A generation unit that generates vibration distribution information showing the vibration distribution of the object based on the velocity information and the three-dimensional recognition information, A detection unit that detects abnormalities in the object based on the vibration distribution information, Equipped with, The generating unit is Based on the point cloud corresponding to the object, extracted as three-dimensional information from the point cloud based on the velocity information of the point cloud and the three-dimensional recognition information, vibrations in the optical axis direction of the light transmitted by the optical transmitting unit of the object are detected, and a first vibration distribution information is generated which is included in the vibration distribution information and shows the distribution of the vibrations in the optical axis direction on a plane intersecting the optical axis direction. The point cloud further includes brightness information indicating the brightness of each of the plurality of points included in the point cloud, The generating unit is Based on the luminance information and the three-dimensional recognition information, the point cloud corresponding to the object is extracted as two-dimensional information from the point cloud, and vibrations in the field of view direction of the optical detection distance measuring unit are detected, and a second vibration distribution information is generated, which is included in the vibration distribution information and shows the distribution of vibrations in the field of view direction. The detection unit, Based on the first vibration distribution information and the second vibration distribution information generated by the generation unit, abnormality detection of the target object is performed. Information processing device.

2. The generating unit is Based on the velocity information and three-dimensional recognition information of the point cloud, a third vibration distribution information is generated, which is included in the vibration distribution information and indicates the distribution of vibrations in the field of view of the optical detection distance measuring unit, based on the point cloud corresponding to the object, which is extracted as three-dimensional information from the point cloud. The detection unit, Based on the first vibration distribution information, the second vibration distribution information, and the third vibration distribution information generated by the generation unit, abnormality detection of the target object is performed. The information processing apparatus according to claim 1.

3. A first recognition unit that includes an optical transmission unit that transmits light modulated by a frequency continuous modulation wave, and an optical reception unit that receives light and outputs a received signal, and performs recognition processing based on the point cloud output by an optical detection distance measuring unit that outputs a point cloud including a plurality of points each having velocity information based on the received signal, and outputs three-dimensional recognition information of an object, A generation unit that generates vibration distribution information showing the vibration distribution of the object based on the velocity information and the three-dimensional recognition information, A detection unit that detects abnormalities in the object based on the vibration distribution information, A second recognition unit performs recognition processing based on the captured image output by an image sensor that outputs a captured image containing color information based on the received light, and outputs two-dimensional recognition information of the object. Equipped with, The generating unit is Based on the velocity information, the captured image, the three-dimensional recognition information, and the two-dimensional recognition information, the vibration distribution information of the object is generated. The generating unit is Based on the velocity information of the point cloud and the three-dimensional recognition information, the point cloud corresponding to the object is extracted as three-dimensional information from the point cloud, and vibrations in the optical axis direction of the light transmitted by the optical transmitting unit of the object are detected, and the first vibration distribution information included in the vibration distribution information shows the distribution of the vibrations in the optical axis direction on a plane intersecting the optical axis direction; Based on the luminance information indicating the brightness of each of the multiple points included in the point cloud and the three-dimensional recognition information, a second vibration distribution information is included in the vibration distribution information, which is extracted as two-dimensional information from the point cloud corresponding to the object, based on the point cloud, and based on the vibration in the field of view direction of the optical detection distance measuring unit, the vibration in the field of view direction is detected, and the distribution of vibration in the field of view direction is shown; Based on the velocity information of the point cloud and the three-dimensional recognition information, a third vibration distribution information is included in the vibration distribution information, which is extracted as three-dimensional information from the point cloud corresponding to the object, and based on the point cloud corresponding to the object, vibrations in the field of view direction of the optical detection distance measuring unit are detected, and the distribution of vibrations in the field of view direction is shown; Based on the two-dimensional recognition information output by the second recognition unit, a fourth vibration distribution information indicating the vibration distribution in the field of view direction is obtained; Generate, The detection unit, Anomaly detection of the object is performed using the first vibration distribution information, the second vibration distribution information, the third vibration distribution information, and the fourth vibration distribution information. Information processing device.

4. The generating unit is Based on the resolution of the point cloud output from the optical detection distance measuring unit and the resolution of the image output from the image sensor, the system selects which of the second vibration distribution information and the fourth vibration distribution information to use as the vibration distribution information in the field of view direction. The information processing apparatus according to claim 3.

5. The generating unit is When the resolution of the point cloud output from the optical detection distance measuring unit is higher than the resolution of the image output from the image sensor, the second vibration distribution information is selected. The fourth vibration distribution information is selected when the resolution of the image output from the image sensor is higher than the resolution of the point cloud output from the optical detection distance measuring unit. The information processing apparatus according to claim 4.

6. The first recognition unit and the second recognition unit are Based on the target region set for the captured image output by the image sensor, the recognition process is performed accordingly. The information processing apparatus according to claim 3.

7. The aforementioned target area is, Based on the region of interest set for the aforementioned captured image, The information processing apparatus according to claim 6.

8. The system further includes a scanning control unit that generates a scanning control signal to control the scanning range of light transmitted by the optical transmitting unit, The detection unit, Based on the vibration distribution information generated by the generation unit using the point cloud acquired from the scanning range controlled by the scanning control signal, abnormality detection of the target object is performed. The information processing apparatus according to claim 1 or 3.

9. The first recognition unit is, Based on the point cloud output by the optical detection distance measuring unit, the recognition process is performed on the point cloud that is included in the target region set for that point cloud. The information processing apparatus according to claim 1 or 3.

10. The aforementioned target area is, Of the point cloud output by the optical detection distance measuring unit, a region of interest is set based on the point cloud, The information processing apparatus according to claim 9.

11. A first recognition step includes an optical transmission unit that transmits light modulated by a frequency continuous modulation wave, an optical reception unit that receives light and outputs a received signal, and an optical detection distance measuring unit that outputs a point cloud containing a plurality of points each having velocity information based on the received signal, and performs recognition processing based on the point cloud output and outputs three-dimensional recognition information of the object, A generation step of generating vibration distribution information that shows the vibration distribution of the object based on the velocity information and the three-dimensional recognition information, A detection step that detects an abnormality in the object based on the vibration distribution information, It has, The generation step is, Based on the point cloud corresponding to the object, extracted as three-dimensional information from the point cloud based on the velocity information of the point cloud and the three-dimensional recognition information, vibrations in the optical axis direction of the light transmitted by the optical transmitting unit of the object are detected, and a first vibration distribution information is generated which is included in the vibration distribution information and shows the distribution of the vibrations in the optical axis direction on a plane intersecting the optical axis direction. The point cloud further includes brightness information indicating the brightness of each of the plurality of points included in the point cloud, The generation step is, Based on the luminance information and the three-dimensional recognition information, the point cloud corresponding to the object is extracted as two-dimensional information from the point cloud, and vibrations in the field of view direction of the optical detection distance measuring unit are detected, and a second vibration distribution information is generated, which is included in the vibration distribution information and shows the distribution of vibrations in the field of view direction. The aforementioned detection step is, Based on the first vibration distribution information and the second vibration distribution information generated in the generation step, abnormality detection of the object is performed. Information processing methods.

12. A first recognition step comprising: an optical transmitting unit that transmits light modulated by a frequency continuous modulated wave; an optical receiving unit that receives light and outputs a received signal; an optical detection distance measuring unit that outputs a point cloud including a plurality of points each having velocity information based on the received signal; and a first recognition step of performing recognition processing based on the point cloud output by the optical detection distance measuring unit and outputting three-dimensional recognition information of an object; A generation step of generating vibration distribution information that shows the vibration distribution of the object based on the velocity information and the three-dimensional recognition information, A detection step that detects an abnormality in the object based on the vibration distribution information, A second recognition step involves performing recognition processing based on the captured image output by an image sensor that outputs a captured image containing color information based on the received light, and outputting two-dimensional recognition information of the object. Equipped with, The generation step is, Based on the velocity information, the captured image, the three-dimensional recognition information, and the two-dimensional recognition information, the vibration distribution information of the object is generated. The generation step is, Based on the velocity information of the point cloud and the three-dimensional recognition information, the point cloud corresponding to the object is extracted as three-dimensional information from the point cloud, and vibrations in the optical axis direction of the light transmitted by the optical transmitting unit of the object are detected, and the first vibration distribution information included in the vibration distribution information shows the distribution of the vibrations in the optical axis direction on a plane intersecting the optical axis direction; Based on the luminance information indicating the brightness of each of the multiple points included in the point cloud and the three-dimensional recognition information, a second vibration distribution information is included in the vibration distribution information, which is extracted as two-dimensional information from the point cloud corresponding to the object, based on the point cloud, and based on the vibration in the field of view direction of the optical detection distance measuring unit, the vibration in the field of view direction is detected, and the distribution of vibration in the field of view direction is shown; Based on the velocity information of the point cloud and the three-dimensional recognition information, a third vibration distribution information is included in the vibration distribution information, which is extracted as three-dimensional information from the point cloud corresponding to the object, and based on the point cloud corresponding to the object, vibrations in the field of view direction of the optical detection distance measuring unit are detected, and the distribution of vibrations in the field of view direction is shown; Based on the two-dimensional recognition information output by the second recognition step, a fourth vibration distribution information showing the vibration distribution in the field of view direction is obtained; Generate, The aforementioned detection step is, Anomaly detection of the object is performed using the first vibration distribution information, the second vibration distribution information, the third vibration distribution information, and the fourth vibration distribution information. Information processing methods.

13. An optical transmission unit that transmits light modulated by a frequency continuous modulation wave, and an optical reception unit that receives light and outputs a received signal, and an optical detection distance measuring unit that outputs a point cloud including a plurality of points each having velocity information based on the received signal, A first recognition unit performs recognition processing based on the point cloud output by the optical detection distance measuring unit and outputs three-dimensional recognition information of the object, A generation unit that generates vibration distribution information showing the vibration distribution of the object based on the velocity information and the three-dimensional recognition information, A detection unit that detects abnormalities in the object based on the vibration distribution information, Equipped with, The generating unit is Based on the point cloud corresponding to the object, extracted as three-dimensional information from the point cloud based on the velocity information of the point cloud and the three-dimensional recognition information, vibrations in the optical axis direction of the light transmitted by the optical transmitting unit of the object are detected, and a first vibration distribution information is generated which is included in the vibration distribution information and shows the distribution of the vibrations in the optical axis direction on a plane intersecting the optical axis direction. The point cloud further includes brightness information indicating the brightness of each of the plurality of points included in the point cloud, The generating unit is Based on the luminance information and the three-dimensional recognition information, the point cloud corresponding to the object is extracted as two-dimensional information from the point cloud, and vibrations in the field of view direction of the optical detection distance measuring unit are detected, and a second vibration distribution information is generated, which is included in the vibration distribution information and shows the distribution of vibrations in the field of view direction. The detection unit, Based on the first vibration distribution information and the second vibration distribution information generated by the generation unit, abnormality detection of the target object is performed. Sensing system.

14. An optical transmission unit that transmits light modulated by a frequency continuous modulation wave, and an optical reception unit that receives light and outputs a received signal, and an optical detection distance measuring unit that outputs a point cloud including a plurality of points each having velocity information based on the received signal, A first recognition unit performs recognition processing based on the point cloud output by the optical detection distance measuring unit and outputs three-dimensional recognition information of the object, A generation unit that generates vibration distribution information showing the vibration distribution of the object based on the velocity information and the three-dimensional recognition information, A detection unit that detects abnormalities in the object based on the vibration distribution information, A second recognition unit performs recognition processing based on the captured image output by an image sensor that outputs a captured image containing color information based on the received light, and outputs two-dimensional recognition information of the object. Equipped with, The generating unit is Based on the velocity information, the captured image, the three-dimensional recognition information, and the two-dimensional recognition information, the vibration distribution information of the object is generated. The generating unit is Based on the velocity information of the point cloud and the three-dimensional recognition information, the point cloud corresponding to the object is extracted as three-dimensional information from the point cloud, and vibrations in the optical axis direction of the light transmitted by the optical transmitting unit of the object are detected, and the first vibration distribution information included in the vibration distribution information shows the distribution of the vibrations in the optical axis direction on a plane intersecting the optical axis direction; Based on the luminance information indicating the brightness of each of the multiple points included in the point cloud and the three-dimensional recognition information, a second vibration distribution information is included in the vibration distribution information, which is extracted as two-dimensional information from the point cloud corresponding to the object, based on the point cloud, and based on the vibration in the field of view direction of the optical detection distance measuring unit, the vibration in the field of view direction is detected, and the distribution of vibration in the field of view direction is shown; Based on the velocity information of the point cloud and the three-dimensional recognition information, a third vibration distribution information is included in the vibration distribution information, which is extracted as three-dimensional information from the point cloud corresponding to the object, and based on the point cloud corresponding to the object, vibrations in the field of view direction of the optical detection distance measuring unit are detected, and the distribution of vibrations in the field of view direction is shown; Based on the two-dimensional recognition information output by the second recognition unit, a fourth vibration distribution information indicating the vibration distribution in the field of view direction is obtained; Generate, The detection unit, Anomaly detection of the object is performed using the first vibration distribution information, the second vibration distribution information, the third vibration distribution information, and the fourth vibration distribution information. Sensing system.

Images