Monitoring system and monitoring method

Abstract

To provide a monitoring system and a monitoring method which enable a user to appropriately determine a situation by using a sensor capable of detecting a distance and a shape.SOLUTION: The system comprises: the sensor capable of detecting a distance and a shape; identification processing means which acquires point group information from the sensor and identifies a detected object as a previously determined type; and superimposing means which selects an image or animation corresponding to the type of the detected object identified by the identification processing means and superimposes the image or animation on a position corresponding to the point group information of the detected object.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a monitoring system and a monitoring method, and more particularly to a monitoring system and a monitoring method using a sensor capable of detecting distance and shape. [Background technology]

[0002] LiDAR sensors are sensors that can detect the distance to an object and its shape. LiDAR sensors can recognize objects even in extremely low-light environments, such as at night, where it is generally difficult for visible cameras to recognize objects. LiDAR sensors can acquire point cloud information of objects and recognize objects from that point cloud information. In this case, they can be linked to a visible camera. For example, by performing a process to control the orientation of a visible camera relative to the subject detected by the LiDAR sensor, the information from the visible camera can be confirmed against the point cloud information acquired by the LiDAR. Without a visible camera, information notification will only be provided in the form of text analyzed based on the point cloud information.

[0003] Patent document 1 also discloses a spatial detection system that includes a distance image sensor unit that generates a distance image including depth information for a shooting range that includes the detection space, a detection plane setting unit that stores spatial position information in three-dimensional space for a specified detection plane within the detection space, a spatial detection unit that detects a ``detection target entering from the detection plane'' by selecting the distance image based on the spatial position information of the detection plane, and an output unit that outputs detection information for the detection target detected by the spatial detection unit. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2021 / 059385 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when LiDAR sensors are used in conjunction with visible light cameras, there are cases where the subject cannot be properly identified, particularly at night, where visible light cameras are not particularly adept. In such cases, the user may not be able to visually assess the situation. Therefore, even if a LiDAR sensor, which is less susceptible to external factors, is used, the results cannot be properly communicated to the user.

[0006] Furthermore, Patent Document 1 discloses a space detection technique for detecting that a detection target has entered a predetermined space, but the technique has the same problems as those described above.

[0007] In view of the above-mentioned problems, an object of the present invention is to provide a monitoring system and a monitoring method that enable a user to more accurately assess a situation while using a sensor that can detect distance and shape. [Means for solving the problem]

[0008] In order to achieve the above object, one representative surveillance system of the present invention comprises a sensor capable of detecting distance and shape, an identification processing means for acquiring point cloud information from the sensor and performing processing to identify detected objects as predetermined types, and a superimposition processing means for selecting an image or animation corresponding to the type of detected object identified by the identification processing means and superimposing the image or animation at a position corresponding to the point cloud information of the detected object, , distance The number of point cloud information, overall size, movement speed, Both of Based on this, the type of the detected object is determined, and if the type of the detected object cannot be identified because the number of points in the point cloud information is small, the superimposition processing means superimposes an image indicating that the type of the detected object cannot be identified. When the number of points in the point cloud increases and the type of detected object can be identified, an image or animation corresponding to the identified type of detected object is superimposed. It is characterized by:

[0009] Furthermore, one of the monitoring methods of the present invention is a monitoring method in which processing is performed by a computer, and includes the steps of: acquiring point cloud information from a sensor capable of detecting distance and shape; performing processing to identify detected objects as predetermined types from the acquired point cloud information; and performing processing to select an image or animation corresponding to the identified type of detected object and superimpose it at a position corresponding to the point cloud information of the detected object, wherein the step of performing processing to identify the detected object is , distance The number of point cloud information, overall size, movement speed, Both of Based on this, the step of determining the type of the detected object and performing the superimposing process includes superimposing an image indicating that the type of the detected object cannot be identified if the number of points in the point cloud information is small and the type of the detected object cannot be identified. When the number of points in the point cloud increases and the type of detected object can be identified, an image or animation corresponding to the identified type of detected object is superimposed. It is characterized by: [Effects of the Invention]

[0010] According to the present invention, in a monitoring system and a monitoring method, a sensor capable of detecting distance and shape is used, allowing a user to more accurately assess a situation. Problems, configurations, and effects other than those described above will become apparent from the following embodiments. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram of a computer system for implementing aspects according to an embodiment of the present disclosure. [Figure 2] FIG. 2 shows a block diagram illustrating one embodiment of the monitoring system of the present invention. [Figure 3] Figure 3 shows (A) an example of a visible camera display and (B) an example of a point cloud display from a LiDAR sensor. [Figure 4] FIG. 4 shows an example of a flowchart of the monitoring system of the present invention. [Figure 5] FIG. 5 shows an example of the processing flow of the monitoring system of the present invention. [Figure 6]FIG. 6 is a diagram illustrating an example of an image display of the surveillance system of the present invention, where (A) shows an image taken by a visible light camera in a dark environment, (B) shows an image detected at a long distance, and (C) shows an image detected at a close distance. DETAILED DESCRIPTION OF THE INVENTION

[0012] An embodiment of the present invention will be described.

[0013] <Computer system for implementing aspects according to the embodiment> 1 is a block diagram of a computer system 1 for implementing aspects according to an embodiment of the present disclosure. The mechanisms and devices of various embodiments disclosed herein may be applied to any suitable computing system. Major components of computer system 1 include one or more processors 2, memory 4, terminal interface 12, storage interface 14, I / O (input / output) device interface 16, and network interface 18. These components may be interconnected via a memory bus 6, an I / O bus 8, a bus interface unit 9, and an I / O bus interface unit 10.

[0014] Computer system 1 may include one or more processing units 2A and 2B, collectively referred to as processors 2. Each processor 2 executes instructions stored in memory 4 and may include an on-board cache. In some embodiments, computer system 1 may include multiple processors, while in other embodiments, computer system 1 may be a single processing unit. Examples of processing units include a central processing unit (CPU), a field-programmable gate array (FPGA), a graphics processing unit (GPU), and a digital signal processor (DSP).

[0015] In some embodiments, memory 4 may include random-access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing data and programs. In some embodiments, memory 4 represents the entire virtual memory of computer system 1 and may include virtual memories of other computer systems connected to computer system 1 via a network. While memory 4 may be conceptually considered a single entity, in other embodiments, memory 4 may be a more complex organization, such as a hierarchy of caches and other memory devices. For example, memory may exist as multiple levels of caches, and these caches may be divided by function. As a result, one cache may hold instructions, while other caches hold non-instruction data used by the processor. Memory may also be distributed and associated with various different processing units, such as in a so-called NUMA (Non-Uniform Memory Access) computer architecture.

[0016] Memory 4 may store all or part of the programs, modules, and data structures that implement the functions described herein. For example, memory 4 may store latent factor identification application 50. In some embodiments, latent factor identification application 50 may include instructions or descriptions that execute the functions described below on processor 2, or may include instructions or descriptions that are interpreted by other instructions or descriptions. In some embodiments, latent factor identification application 50 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices instead of or in addition to a processor-based system. In some embodiments, latent factor identification application 50 may include data other than instructions or descriptions. In some embodiments, cameras, sensors, or other data input devices (not shown) may be provided to communicate directly with bus interface unit 9, processor 2, or other hardware in computer system 1. Such a configuration may reduce the need for processor 2 to access memory 4 and the latent factor identification application.

[0017] Computer system 1 may include a bus interface unit 9 that facilitates communication between processor 2, memory 4, display system 24, and I / O bus interface unit 10. I / O bus interface unit 10 may couple to I / O bus 8 for transferring data to and from various I / O units. I / O bus interface unit 10 may communicate with multiple I / O interface units 12, 14, 16, and 18, also known as I / O processors (IOPs) or I / O adapters (IOAs), via I / O bus 8. Display system 24 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both data to display device 26. Computer system 1 may also include one or more sensors or other devices configured to collect data and provide the data to processor 2. For example, computer system 1 may include environmental sensors that collect humidity data, temperature data, pressure data, etc., and motion sensors that collect acceleration data, movement data, etc. Other types of sensors may also be used. Display memory may be dedicated memory for buffering video data. The display system 24 may be connected to a display device 26, such as a standalone display screen, a television, a tablet, or a handheld device. In some embodiments, the display device 26 may include a speaker for rendering audio. Alternatively, the speaker for rendering audio may be connected to an I / O interface unit. In other embodiments, the functionality provided by the display system 24 may be implemented by an integrated circuit that includes the processor 2. Similarly, the functionality provided by the bus interface unit 9 may be implemented by an integrated circuit that includes the processor 2.

[0018] The I / O interface unit provides functionality for communicating with various storage or I / O devices. For example, the terminal interface unit 12 may be equipped with user I / O devices 20, such as user output devices such as a video display, a television with speakers, and user input devices such as a keyboard, a mouse, a keypad, a touchpad, a trackball, buttons, a light pen, or other pointing device. A user may use a user interface to enter input data or instructions into the user I / O devices 20 and the computer system 1 and receive output data from the computer system 1 by operating the user input devices. The user interface may be displayed on a display, played through speakers, or printed via a printer via the user I / O devices 20, for example.

[0019] Storage interface 14 may be attached to one or more disk drives or direct access storage devices 22 (typically magnetic disk drive storage devices, but may also be an array of disk drives or other storage devices configured to appear as a single disk drive). In some embodiments, storage device 22 may be implemented as any secondary storage device. The contents of memory 4 may be stored in storage device 22 and retrieved from storage device 22 as needed. Network interface 18 may provide a communications path that allows computer system 1 and other devices to communicate with each other. This communications path may be, for example, a network 30.

[0020] While computer system 1 shown in FIG. 1 includes a bus structure providing direct communication paths between processor 2, memory 4, bus interface 9, display system 24, and I / O bus interface unit 10, in other embodiments, computer system 1 may include point-to-point links, multiple hierarchical buses, parallel or redundant communication paths in a hierarchical, star, or web configuration. Furthermore, while I / O bus interface unit 10 and I / O bus 8 are shown as a single unit, computer system 1 may actually include multiple I / O bus interface units 10 or multiple I / O buses 8. Additionally, while multiple I / O interface units are shown isolating I / O bus 8 from various communication paths leading to various I / O devices, in other embodiments, some or all of the I / O devices may be directly connected to a single system I / O bus.

[0021] In some embodiments, computer system 1 may be a device that receives requests from other computer systems (clients) without a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. In other embodiments, computer system 1 may be a desktop computer, a portable computer, a laptop, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device.

[0022] <Block diagram of monitoring system> FIG. 2 shows a block diagram illustrating one embodiment of the monitoring system of the present invention.

[0023] The monitoring system 100 includes a sensor 101, a switch 103, an identification processing server 104, a superimposition processing server 105, and a monitoring terminal 106. Furthermore, a visible camera 102 can be included as needed.

[0024] The sensor 101 is a sensor that can detect distance and shape. In particular, it is preferable to apply a LiDAR (Light Detection And Ranging) sensor. Here, the LiDAR sensor emits laser light, measures the time it takes for the light to hit an object and bounce back, and measures the distance and direction to the object. In this case, it is possible to detect not only the distance to the object but also its position and shape from point cloud information. The higher the detection accuracy of the sensor 101 (the more points in the point cloud), the better. The sensor 101 performs measurements in real time, and sends measurement results to the identification processing server 104 multiple times per second or more.

[0025] The sensor 101 is installed in a location or area to be monitored. Multiple sensors 101 may be provided and installed in different locations. Furthermore, the multiple sensors 101 may perform sensing from different directions. The sensor 101 can detect an object or intruder that is not actually present in that location and issue an alarm.

[0026] The switch 103 is installed between the sensor 101, the visible light camera 102, the identification processing server 104, the superimposition processing server 105, and the monitoring terminal 106. Information about an object detected by the sensor 101 is transmitted to the identification processing server 104 via the switch 103. Furthermore, information processed by the identification processing server 104 is transmitted to the superimposition processing server 105 via the switch 103. Furthermore, information processed by the superimposition processing server 105 is transmitted to the monitoring terminal 106 via the switch 103. Furthermore, information about an object detected by the visible light camera 102 is transmitted to the identification processing server 104, the superimposition processing server 105, or the monitoring terminal 106 via the switch 103, as necessary. Here, this information may be transmitted via a wired or wireless connection, or via a network such as the Internet.

[0027] The identification processing server 104 performs object recognition based on information about the object detected by the sensor 101. This processing can be performed using AI (artificial intelligence). For example, this can be performed using a trained model obtained by machine learning based on point cloud information about the object from the sensor 101. At this time, for example, a neural network model or the like is used. This allows object recognition to be performed and the type of object to be identified. The type of object at this time is determined by identifying the type of object itself. Furthermore, the behavior (state) of the object may also be determined.

[0028] The superimposition processing server 105 performs superimposition processing based on the classification of the object type by the classification processing server 104. The superimposition processing here is processing for superimposing image information, etc. related to the object type that has been set in advance, and a specific example will be described later. The superimposed information is then transmitted to the monitoring terminal 106. This transmission may be performed when an alarm is detected. When superimposing image information, etc., the image information, etc. is converted to an optimal position and size based on the detection distance and movement information related to the object from the classification processing server 104, and display control is performed.

[0029] The identification processing server 104 and the superimposition processing server 105 can cooperate with each other, or may be configured as an integrated unit. The computer system 1 in Fig. 1 can be applied to these. Note that each of the identification processing server 104 and the superimposition processing server 105 may be realized by multiple servers working together.

[0030] The visible light camera 102 may be configured as a camera that obtains information by focusing incident light onto an image sensor via a lens and an aperture. Examples of the image sensor include a CCD (Charge-Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The visible light camera 102 captures images at, for example, three frames per second (3 fps) or more. The visible light camera 102 may capture images of the same range as the sensor 101. Furthermore, upon detecting an alarm, the overlay processing server 105 may control the visible light camera 102 to point toward the alarm point.

[0031] The monitoring terminal 106 has a display function that can display the content processed by the superposition processing server 105. This display function can be configured, for example, with a liquid crystal display or an organic light-emitting diode (OLED) display. The monitoring terminal 106 may be configured with a touch panel, or may be equipped with operation means such as a keyboard or mouse. The monitoring terminal 106 may have a function that displays results from different sensors 101 by switching the display. The computer system 1 in FIG. 1 can also be applied to the monitoring terminal 106.

[0032] <Example of visible camera and LiDAR sensor display> Figure 3 shows (A) an example of a visible camera display and (B) an example of a point cloud display from a LiDAR sensor.

[0033] Figure 3 compares the display of an object detected in the same direction and size by a visible camera and a LiDAR sensor. Figure 3(A) shows the display by the visible camera, capturing a small animal, cat 201. Figure 3(B) shows the cat 201 detected by the LiDAR sensor, which is detected as point cloud information 202 consisting of multiple points. The point cloud information 202 has the characteristic that the further away an object is, the fewer points there are on the object, and the closer it is, the more points there are on the object. Furthermore, the higher the detection accuracy of the LiDAR sensor, the more point cloud information there is. In Figure 3(B), the analysis results based on the point cloud information 202 are displayed as text information 203. Here, it is identified that "alert in progress" and "small animal" are being detected.

[0034] As shown in FIG. 3(B), when the point cloud information 202 output from the LiDAR sensor is displayed as is, it becomes information of a collection of points. For this reason, it is difficult for a user looking at this point cloud information 202 to accurately determine that it is a cat and its behavioral state, as is the case with cat 201 photographed by a visible light camera. For this reason, it is necessary to display the analysis results as text information 203 or the like on the screen separately from the point cloud information 202 to inform the user of the situation. However, since the user needs to check the text information and, in addition, there may be multiple target objects, it is not possible to make an intuitive judgment.

[0035] <Flowchart> FIG. 4 shows an example of a flowchart of the monitoring system of the present invention.

[0036] First, in step S301, point cloud information is acquired from the sensor 101. An example of the point cloud information is shown in Fig. 3. The acquired point cloud information is sent to the identification processing server 104.

[0037] Next, in step S302, the point cloud information is analyzed. This analysis is performed by the identification processing server 104. In the analysis, an object to be detected is detected and the type of the object is identified.

[0038] The detection of a target object can be performed by detecting a cluster of point cloud information that can be identified as an object, and then comparing this cluster with predetermined conditions to determine whether it exhibits different movement. For example, this can occur when a moving object enters the detection range of a still image, or when an object moves in a different direction, such as the opposite direction, compared to a state in which an object is moving in a fixed direction. In this case, an alarm can be triggered.

[0039] The type of object being detected can be identified using AI as described above, and in addition to the type of object itself, the state of the object may also be determined. The type of object is determined based on the type required for the intended monitoring, such as a person, animal, bicycle, or automobile. The state of the object, in the case of a person, is determined based on the state of the object, such as whether the person is walking, running, sitting, or crouching, and this state is also determined based on the state required for the intended monitoring. These determinations can be made based on the size, shape, reflectivity, and movement state (amount of movement) of the point cloud information of the target object.

[0040] Next, in step S303, tagging is performed. Tagging is performed by the identification processing server 104. The type of object determined in step S302 is specified and tagged for a block of point cloud information that can be identified as an object. For example, "object type is person, object state is walking state." The type of object is maintained as the same object tag even if the block of point cloud information that can be identified as an object moves. Furthermore, the tag information for the state of the object may be changed depending on the situation. Furthermore, it is sufficient to tag only the types necessary for monitoring, and tagging may not be performed if it is determined that it is not necessary. In this case, the type can be registered in advance to make the determination. This tag registration may be selectable by user operation on the monitoring terminal 106.

[0041] Next, in step S304, an image or the like is superimposed. This process is performed by the superimposition processing server 105. For a block of point cloud information that can be identified as an object, the superimposition processing server 105 selects an image or animation corresponding to the tag and superimposes it at a position corresponding to the point cloud information. By superimposing, the image or animation is lightly superimposed on the point cloud information, or the point cloud information is replaced with the image or animation. For example, if the object type is a person and the object state is walking, an image or animation of a walking person is superimposed on the corresponding point cloud information. The image or animation is pre-stored in the superimposition processing server 105 or the like according to the type of tag. The image or animation is a picture, photograph, animation, or the like whose type can be visually identified by a person. The size of the image or animation to be superimposed is adjusted to fit the range of the corresponding point cloud information. A specific example will be described later with reference to FIG. 6.

[0042] Here, several superimposition patterns are possible. The first is a pattern in which a selected image or animation is superimposed on an image of the sensor 101. Because the image of the sensor 101 is represented by point cloud information, an image or animation is superimposed on the portion of the point cloud information of the corresponding object instead of the point cloud information. In this case, the image of the sensor 101 is used, thereby reducing the processing load. The second is a pattern in which a still image corresponding to the measurement range of the sensor 101 is prepared, and a selected image or animation is superimposed at a corresponding position on the still image. In this case, the still image serves as a visual background image, making it easier for the user to recognize the situation. The third is a pattern in which, when a visible camera 102 corresponding to the measurement range of the sensor 101 is used, the selected image or animation is superimposed at a corresponding position on the screen of the visible camera 102. In this case, the corresponding positions can be aligned by, for example, aligning the coordinates of the sensor 101 and the visible camera 102. This makes it possible to identify an alarm-activating object in the image of the visible camera 102, even in dark conditions.

[0043] Next, in step S305, the superimposed data is output from the superimposition processing server 105 to the monitoring terminal 106. The monitoring terminal 106 displays the superimposed information on its display function so that the user can check it.

[0044] <An example of the processing flow of the monitoring system> Fig. 5 shows an example of the processing flow of the monitoring system of the present invention. Fig. 5 explains an example in which a person 401 and a small animal 402 are detected by the sensor 101. Note that this shows an example in which only the type of object is taken into consideration, and the state of the object is not taken into consideration.

[0045] If it is a person 401, first, the sensor 101 acquires point cloud information of the person 401. Next, the identification processing server 104 determines that it is a person based on the point cloud information. This determination is made based on the number of point cloud information pieces for a specific distance, the overall size, the movement speed, etc. For this reason, the point cloud information of the sensor 101 within a predetermined time range is also taken into account. If the identification processing server 104 determines that it is a person, the tag is set to "person." Next, the superimposition processing server 105 performs a process of superimposing image A (an image of a person) corresponding to the tag "person" on the position of the point cloud information. The superimposed information is displayed on the monitoring terminal 106.

[0046] If it is a small animal 402, first, the sensor 101 acquires point cloud information of the small animal 402. Next, the identification processing server 104 determines that it is a small animal based on the point cloud information. This determination is made based on the small number of point cloud information for a specific distance, the overall size, the movement speed, etc. For this reason, the point cloud information of the sensor 101 within a predetermined time range is taken into consideration. If the identification processing server 104 determines that it is a small animal, the tag is set to "small animal." Next, the superimposition processing server 105 performs a process of superimposing image B (an image of the small animal) corresponding to the tag "small animal" on the position of the point cloud information. The superimposed information is displayed on the monitoring terminal 106.

[0047] <Example of image display from a surveillance system> 6A and 6B are diagrams illustrating examples of image displays of the monitoring system of the present invention, where (A) shows an image taken by a visible camera in dark conditions, (B) shows a case where an object is detected at a long distance, and (C) shows a case where an object is detected at a short distance. Fig. 6A is an example of an image taken by the visible camera 102 displayed on the monitoring terminal 106. Figs. 6B and 6C show examples of information output from the superposition processing server 105 displayed on the monitoring terminal 106.

[0048] As shown in Figure 6(A), it is difficult to visually identify an object (a person in this case) in a dark image captured by a visible light camera. For this reason, as explained above, the superimposition processing server 105 superimposes image information on the detected object and displays that information on the screen of the monitoring terminal 106. When displaying the superimposed image information, the display size is optimized based on information such as the position of the subject.

[0049] By performing optimization, when the sensor 101 detects an object at a long distance, as shown in Fig. 6(B), processing is performed to display image C in a small size. Also, when the sensor 101 detects an object at a short distance, as shown in Fig. 6(C), processing is performed to display image C in a large size.

[0050] The following methods can be used as optimization methods. One is to determine the size based on the range of the point cloud of the target object. For example, if the range of the point cloud of the target object is large, the size of the image or animation to be superimposed is increased to match that size. The other is to adjust the size using perspective based on the distance, since the sensor 101 is a sensor that can measure the distance to the object. Optimization may also be performed by combining these two methods.

[0051] Furthermore, if the object is far away, the number of points in the point cloud detected by the sensor 101 is small, and so the type may not be identified. In this case, an image or other display indicating that the type cannot be identified may be superimposed. For example, a "?" mark may be superimposed. Furthermore, when the object approaches the sensor 101 and the number of points in the point cloud increases, the type of the object can be identified, and so an image or animation corresponding to the type can be superimposed.

[0052] Next, a case will be described in which, when an object is nearby and its type is identified and an image or animation corresponding to that type is superimposed, the object moves away from the sensor 101. In this case, the number of points in the point cloud decreases along the way, and it may become impossible to identify the type from the point cloud information. In this case, if it is determined that the block of point cloud information is moving continuously without interruption, the image or animation that was superimposed up until then may remain superimposed while optimizing its size.

[0053] <effect> If the user can monitor using the visible camera 102, the monitoring terminal 106 displays the image from the visible camera 102. However, if the image cannot be seen from the visible camera 102 or if the system does not include a visible camera 102, the information based on the sensor 101 described above is displayed on the screen of the monitoring terminal 106.

[0054] Depending on the conditions, it may also be possible to display an image or animation as an overlay on the video from the visible light camera 102. This is the third overlay pattern described in step S304 of FIG. 4. In this case, the mode may be automatically switched when it gets dark by detecting the ambient brightness. Alternatively, a switch or switch display for switching to night mode may be provided and the user may operate it. In this case, a function may be used to display the background brightly or to display the background during bright times such as daytime.

[0055] <Specific application examples> A specific application example will be explained. A sensor 101 such as a LiDAR sensor is installed in a monitoring location or target area to detect intruders and other objects. The detection conditions are basically determined from point cloud information from the sensor 101, along with various parameters such as the size, amount of movement, and reflectivity of the object. The object is then determined through AI processing by the identification processing server 104. Here, multiple conditions can be combined to perform multiple object recognition and determination. For example, several classifications such as people, animals, and cars are performed. Classification is applied according to pre-defined conditions according to the user's requests.

[0056] On the application of the superimposition processing server 105, an image of the classified type is displayed on the screen at the same time as an alarm is detected. If the detected object is a moving object such as an intruder, the image may be animated or moved. Also, optimal information for the user may be displayed on the screen by performing processing and control to change the display size of the image to match the actual object based on the detection distance and movement information.

[0057] <Effects> According to the above-described embodiment, the identification processing server 104 determines whether an object has been detected and recognized from the point cloud information from the sensor 101, and the superimposition processing server 105 superimposes information such as an image. This makes it possible to display the information on the monitoring terminal 106 used by the user in a state in which the object can be recognized and judged. For users accustomed to general monitoring cameras, it is difficult to judge an alarm if only text information is displayed on the terminal, but the above-described configuration makes it possible to provide optimal information to the user. Furthermore, the superimposed image, etc. can be displayed in an easy-to-understand manner by optimizing the size.

[0058] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0059] For example, the sensor 101 is the LiDAR sensor described above, which is suitable for measuring long distances. However, the sensor 101 can be applied to other sensors as long as they can detect distance and shape, and for example, a TOF (Time of Flight) method may be used. [Explanation of symbols]

[0060] 1...computer system, 2...processor, 2A, 2B...processing device, 4...memory, 6...memory bus, 8...I / O bus, 9...bus interface unit, 10...I / O bus interface unit, 12...terminal interface unit, 14...storage interface, 16...I / O device interface, 18...network interface, 20...user I / O device, 22...storage device, 24...display system, 26...display device, 30...network, 50...latent factor identification application, 100...surveillance system, 101...sensor, 102...visible camera, 103...switch, 104...identification processing server, 105...superimposition processing server, 106...surveillance terminal, 201...cat, 202...point cloud information, 203...text information, 401...person, 402...small animal

Claims

1. The system comprises a sensor capable of detecting distance and shape, an identification processing means for acquiring point cloud information from the sensor and performing processing to identify detected objects as predetermined types, and a superimposition processing means for selecting an image or animation corresponding to the type of detected object identified by the identification processing means and superimposing the image or animation at a position corresponding to the point cloud information of the detected object, The identification processing means determines the type of detected object based on the number of points, the overall size, and the moving speed of the point cloud information relative to the distance; The superimposition processing means is characterized in that, when the type of detected object cannot be identified because the number of points in the point cloud information is small, it superimposes an image indicating that the type of detected object cannot be identified, and when the number of points in the point cloud increases and the type of detected object can be identified, it superimposes an image or animation corresponding to the identified type of detected object.

2. 2. The monitoring system according to claim 1, A monitoring system characterized in that the sensor is a LiDAR sensor.

3. 2. The monitoring system according to claim 1, A surveillance system characterized in that the type of detected object is identified by the identification processing means by distinguishing it into a predetermined object type and tagging it with that type.

4. 2. The monitoring system according to claim 1, A surveillance system characterized in that the identification processing means identifies detected objects using AI that uses a trained model.

5. 2. The monitoring system according to claim 1, The monitoring system is characterized in that the superimposition processing means combines a background image of the same range as the range detected by the sensor as a background of the point cloud information from the sensor.

6. 2. The monitoring system according to claim 1, The superimposition processing means determines the size of the image or animation to be superimposed on the point cloud information of the detected object based on at least one of the distance acquired by the sensor and the size of the point cloud information of the detected object. A monitoring system characterized by being optimized using both.

7. 2. The monitoring system according to claim 1, A surveillance system characterized in that, if the superimposition processing means determines that the block of point cloud information of the detected object is moving continuously without interruption, it performs processing to keep the image or animation that was superimposed up to that point superimposed while optimizing its size.

8. 2. The monitoring system according to claim 1, A monitoring system comprising a monitoring terminal, wherein the superimposition processing means outputs information in which the image or animation is superimposed on point cloud information of the detected object to the monitoring terminal.

9. a sensor capable of detecting distance and shape, a visible camera that photographs the detection range of the sensor, an identification processing means that acquires point cloud information from the sensor and identifies detected objects as a predetermined type, and a superimposition processing means that selects an image or animation corresponding to the type of detected object identified by the identification processing means and superimposes it on a position of the image of the visible camera that corresponds to the position of the point cloud information of the detected object, The identification processing means determines the type of detected object based on the number of points, the overall size, and the moving speed of the point cloud information relative to the distance; The superimposition processing means is characterized in that, when the type of detected object cannot be identified because the number of points in the point cloud information is small, it superimposes an image indicating that the type of detected object cannot be identified, and when the number of points in the point cloud increases and the type of detected object can be identified, it superimposes an image or animation corresponding to the identified type of detected object.

10. A monitoring method that performs processing by a computer, acquiring point cloud information from a sensor capable of detecting distance and shape; performing a process of classifying detected objects into predetermined types based on the acquired point cloud information; and performing a process of selecting an image or animation corresponding to the type of the detected object identified and superimposing the image or animation at a position corresponding to the point cloud information of the detected object, The step of performing the identification process includes determining the type of the detected object based on the number of points, the overall size, and the moving speed of the point cloud information relative to the distance; A monitoring method characterized in that the step of performing the superimposition process superimposes an image indicating that the type of detected object cannot be identified when the number of points in the point cloud information is small and the type of detected object cannot be identified, and when the number of points in the point cloud increases and the type of detected object can be identified, superimposes an image or animation corresponding to the identified type of detected object.

Images