Monitoring device, monitoring system, monitoring method, and monitoring program

Abstract

To provide a simple method for setting a monitoring target when using a camera mounted on a vehicle as a surveillance camera.SOLUTION: The monitoring device includes a control unit configured to perform the steps of: accepting a user operation for designating a range in an image of surroundings of a vehicle captured by a camera mounted on the vehicle; and setting the designated range as a monitoring target.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a monitoring device, a monitoring system, a monitoring method, and a monitoring program for using a camera mounted on a vehicle as a monitoring camera. [Background technology]

[0002] It has been proposed to utilize cameras mounted on vehicles as surveillance cameras to monitor the periphery of a parking space (for example, Patent Documents 1 and 2). Patent Document 1 describes changing the detection range of a suspicious person depending on the parking position. Patent Document 2 describes changing the imaging range of each imaging device depending on the number and positions of imaging devices present around the monitored object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-149088 [Patent Document 2] Japanese Patent Application Laid-Open No. 2008-160496 Summary of the Invention [Problem to be solved by the invention]

[0004] When using a vehicle-mounted camera as a surveillance camera, for example, during outdoor activities such as car camping, the items to be monitored and their locations may change. However, in the past, it was cumbersome to set the monitored items each time the camera was used. Therefore, an object of the present disclosure is to provide a simple method for setting the monitored items when using a vehicle-mounted camera as a surveillance camera. [Means for solving the problem]

[0005] The monitoring device according to the present disclosure includes a control unit that accepts a user's operation to specify a range in an image of the surroundings of a vehicle captured by a camera mounted on the vehicle, and sets the specified range as a monitoring target. [Effects of the Invention]

[0006] According to the present technology, when a camera mounted on a vehicle is used as a surveillance camera, it is possible to provide a simple method for setting a surveillance target. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram for explaining an outline of the embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an example of a system configuration. [Figure 3] FIG. 3 is a process flow diagram illustrating an example of the setting process. [Figure 4] FIG. 4 is a diagram for explaining image synthesis. [Figure 5] FIG. 5 is a diagram showing an example of an overhead image. [Figure 6] FIG. 6 is a diagram showing an example of a panoramic image. [Figure 7] FIG. 7 is a processing flow diagram illustrating an example of the monitoring processing. DETAILED DESCRIPTION OF THE INVENTION

[0008] <Embodiment> Hereinafter, an embodiment will be described with reference to the drawings. FIG. 1 is a diagram for explaining an outline of the embodiment. In this embodiment, an in-vehicle device (monitoring device) 10 of a vehicle 1 uses cameras 21-24 mounted on the vehicle 1 as monitoring cameras. The in-vehicle device 10 uses one or more cameras 21-24. The system accepts the user's designation of a monitoring target from the images captured by the camera, and performs processing to monitor the designated target. The image corresponds to one still image among a series of frames that make up a moving image. The image may also be a composite image that combines images captured by multiple cameras and allows the surroundings of the vehicle 1 to be viewed from a desired viewpoint. The composite image may be a three-dimensional bird's-eye view image of the surroundings of the vehicle, or may be a three-dimensional CG (Computer Graphics) image. The composite image may include the vehicle itself drawn by the method described above. Furthermore, the composite image may not be an overhead image, but may be a first-person perspective image such as a panoramic image stitched using the equirectangular method.

[0009] Camera 21 is a front camera that is provided, for example, on the front grille and captures an image in front of vehicle 1. Camera 22 is a side camera that is provided, for example, near the side mirror and captures an image on the left side of vehicle 1. Camera 23 is a side camera that is provided, for example, near the side mirror and captures an image on the right side of vehicle 1. Camera 24 is a rear camera that is provided, for example, on the back door and captures an image behind vehicle 1. As shown in FIG. 1, the in-vehicle device may function as an image synthesis unit 111 that acquires images from four cameras 21-24 and creates a synthesized image.

[0010] The composite image can be displayed as an image of the surroundings of the vehicle 1 viewed from any virtual viewpoint. The in-vehicle device 10 may function as a viewpoint change unit 112 that changes the viewpoint based on, for example, a user operation. The viewpoint change unit 112 sets the position and size of a display range that is part of the composite image (in other words, the magnification ratio at the time of display).

[0011] The in-vehicle device 10 also functions as a monitoring target setting unit 113 that accepts a user's operation to designate a monitoring target and sets the monitoring target. The monitoring target may be designated, for example, by the user swiping (tracing) the periphery of a range on a composite image displayed on a display provided by the in-vehicle device 10 using a touch panel. In this case, the in-vehicle device 10 sets, for example, an area surrounded by the trajectory of the user's swipe as the boundary, as the monitoring target area. The monitoring target may also be designated, for example, by the user sequentially inputting adjacent vertices of a polygon on a composite image displayed on a display provided by the in-vehicle device 10. In this case, the in-vehicle device 10 sets, for example, an area surrounded by a polygon formed by connecting vertices with edges, as the monitoring target area. The monitoring target may also be designated by accepting a user's selection of objects detected from the composite image using an existing object detection method. In this case, the monitoring target is set as an area surrounded by the selected objects or a larger area that surrounds all of the selected objects.

[0012] The in-vehicle device 10 also functions as a monitoring processing unit 114 that monitors a specified monitoring target. In the monitoring process, the monitoring target is monitored based on an existing moving object detection method or an abnormality detection method. For example, the in-vehicle device 10 may detect a moving object or determine the presence or absence of an object by detecting differences between frames constituting the video based on the amount of change in at least one of the hue, brightness, and saturation (hereinafter also referred to as "hue, etc.") of the area of ​​the monitoring target. The monitoring target object may also be tracked by tracking changes in the position of a person or object within a composite image using a pattern matching method such as existing tracking. If the in-vehicle device 10 determines that an abnormality has occurred, it may perform abnormality processing, such as outputting an alarm to the vehicle 1.

[0013] In this way, when using a vehicle-mounted camera as a surveillance camera, the surveillance target can be easily set even if the surveillance target or its location changes each time the camera is used, which means that a simple method for setting the surveillance target can be provided.

[0014] <Device configuration> 2 is a block diagram illustrating an example of the system configuration. The vehicle 1 is, for example, an automobile. The vehicle 1 includes an in-vehicle device 10 and a camera 20 (21-24 in FIG. 1). The in-vehicle device 10 is a computer, and functions as, for example, a surveillance camera in cooperation with the camera 20. The in-vehicle device 10 includes a processor 11, a storage device 12, and a user interface (UI) 13.

[0015] The processor 11 is a processing unit such as a CPU (Central Processing Unit). The processor 11 corresponds to a control unit according to the embodiment. By executing a program, the processor 11 functions as an image synthesis unit 111, a viewpoint change unit 112, a monitoring target setting unit 113, and a monitoring processing unit 114 shown in Fig. 1. The storage device 12 includes a main storage device such as a RAM (Random Access Memory) or a ROM (Read Only Memory), and a hard disk drive (HDD) or a SS (Second-Side Storage). It includes at least one of auxiliary storage devices such as a solid state drive (SD) or flash memory. The storage device 12 temporarily stores programs read by the processor 11 and information to be processed, and secures a working area for the processor 11. The storage device 12 also temporarily or permanently stores image data output by the camera 20 and setting information for the viewpoint and the monitoring target. The UI 13 is a user interface such as a display with a touch panel stacked thereon, a display and physical operation buttons, or a combination of a display, a touch panel and operation buttons. The processor 11 outputs, for example, a composite image via the UI 13, and also receives the designation of the monitoring target. Note that the process of combining images from multiple cameras may be performed by a predetermined ECU (Electronic Control Unit). In other words, the processor The controller 11 and a predetermined ECU may function as the control unit according to this embodiment.

[0016] Camera 20 is an imaging device that converts light into an electrical signal using an image sensor such as a CCD or CMOS, and creates and outputs image data. Camera 20 outputs moving images by repeating the process of creating and outputting images. Note that the focal length of camera 20 is not particularly limited, but in the example of Figure 1, it is desirable for cameras 21-24 to be equipped with a wide-angle lens or a fisheye lens so that they can capture 360-degree images around vehicle 1.

[0017] <Settings process> 3 is a process flow diagram showing an example of a setting process for specifying a monitoring target. The in-vehicle device 10 starts the setting process, for example, in response to a user operation. The camera 20 continuously stores captured image data in the storage device 12 of the in-vehicle device 10. An object detection model for detecting objects from the image data may be stored in advance in the storage device 12. The object detection model can be created by a machine learning method such as deep learning using a convolutional neural network, using objects expected to be monitored as training data.

[0018] The processor 11 of the in-vehicle device 10 acquires image data output by the camera 20 (FIG. 3: step S1). FIG. 4 is a diagram for explaining image synthesis. As shown in FIG. 4(A), the camera 21 provided on the vehicle 1 outputs an image 201 capturing an image in front of the vehicle 1. The camera 22 outputs an image 202 capturing an image of the left side of the vehicle 1. The camera 23 outputs an image 203 capturing an image of the right side of the vehicle 1. The camera 24 outputs an image 204 capturing an image behind the vehicle 1. In step S1, such images 201-204 are acquired. It is assumed that the four images 201-204 can capture a continuous 360-degree image of the area around the vehicle 1.

[0019] After step S1, the processor 11 synthesizes the images (FIG. 3: step S2). In this step, for example, a bird's-eye view image of the surroundings of the vehicle 1 is created. As shown in FIG. 4(B), the processor 11 synthesizes the pixels included in the four images 201-204 in a virtual three-dimensional space. The image is projected onto projection surface 200, which is a curved surface that projects the image. Projection surface 200 is, for example, approximately hemispherical (bowl-shaped). The center of projection surface 200 (bottom of the bowl) corresponds to the position where vehicle 1 is located. The rest of projection surface 200 is associated with one of images 201-204.

[0020] Processor 11 projects images 201-204 onto portions of projection surface 200 other than the center. Processor 11 projects image 201 captured by camera 21 onto an area of ​​projection surface 200 corresponding to the front of vehicle 1, image 202 captured by camera 22 onto an area corresponding to the left side of vehicle 1, image 203 captured by camera 23 onto an area corresponding to the right side of vehicle 1, and image 204 captured by camera 24 onto an area corresponding to the rear of vehicle 1. Note that pixel information may be blended appropriately to smooth the boundaries where the images overlap, or pixel information may be supplemented appropriately for positions on the composite image where no pixels corresponding to images 201-204 exist. In this way, a composite image showing the entire surroundings of vehicle 1 can be created.

[0021] After step S2, the processor 11 displays the composite image on the UI 13 (FIG. 3: step S3). In this step, a portion of the composite image corresponding to the surroundings of the vehicle 1 viewed from a predetermined viewpoint is displayed. It is assumed that information on the initial viewpoint or information on the viewpoint set by the user is stored in advance in the storage device 12. FIG. 5 is a diagram showing an example of an overhead image displayed in step S3. In the example of FIG. 5, outdoor equipment 3 existing around the vehicle 1 is photographed. In step S3, as shown in FIG. 5(A), an image 210 corresponding to a portion of the composite image viewed from a predetermined viewpoint is displayed. It is noted that the vehicle 1 in FIG. 5 is not photographed by the camera 20, but is a CG vehicle placed at the center of the projection surface 200 shown in FIG. 4(B). The overhead image is not limited to one looking down on the vehicle 1 diagonally from above, but may be one looking down on the vehicle 1 from directly above the vehicle 1, for example.

[0022] After S3, the processor 11 determines whether a command to change the viewpoint has been input from the user via the UI 13 (FIG. 3: step S4). The user can perform operations to change the coordinates of the viewpoint, the direction of the line of sight, the magnification rate of the image, etc. via the UI 13. Note that the user may be able to change between a bird's-eye view image and a panoramic image.

[0023] If it is determined in step S4 that an instruction to change the viewpoint has been input (step S4: YES), processor 11 stores information representing the changed viewpoint in storage device 12 (FIG. 3: step S5), and returns to the processing of step S2. That is, a part of the composite image is redisplayed, which corresponds to the surroundings of vehicle 1 viewed from the changed viewpoint.

[0024] On the other hand, if it is determined in step S4 that an instruction to change the viewpoint has not been input (step S4: NO), processor 11 determines whether the user has input a designation of a monitoring target via UI 13 (FIG. 3: step S6). The designation of a monitoring target may be performed, for example, by the user swiping UI 13, which includes a touch panel, on the overhead image shown in FIG. 5A. FIG. 5B shows an example of area 4 designated by swiping. Area 4 is a shape drawn freehand by the user. Note that if area 4 is not closed, a loop may be created by connecting the start point and end point of the line input in one stroke with a straight line or the like, as shown by the dashed line, for example.

[0025] Furthermore, the monitoring target may be specified by the user sequentially inputting adjacent vertices of a polygon on the overhead image shown in (A) of Fig. 5. (C) of Fig. 5 shows an example of an area 5 specified by inputting vertices. Area 5 is a polygon formed by connecting three or more vertices specified by the user. In this case, too, the start point and end point may be connected by a straight line or the like to create a loop, as shown by the dashed line, for example.

[0026] The monitoring target may be designated by receiving a user's selection of an object detected from the composite image by an object detection technique. FIG. 6 shows an example of the panoramic image displayed in step S3. FIG. 6(A) shows a composite image 211 viewed in a predetermined direction from the vehicle 1. FIG. 6(B) shows a composite image 211 in which objects detected by object detection are highlighted. In this example, the outlines of detected objects, such as a backpack, cardboard box, or barbecue grill, are indicated by thick lines. The detected objects may also be displayed surrounded by a bounding box or the like. The user may select an object from the detection results shown in FIG. 6(B) by tapping the area in which the object is captured. Alternatively, an area surrounded by three or more selected objects, or a larger area surrounding all of the selected objects, may be designated as the monitoring target.

[0027] If it is determined in step S6 that a designation of a monitoring target has been input (step S6: YES), the processor 11 stores the monitoring target in the storage device 12 (FIG. 3: step S7). When the monitoring target is designated by an area, a coordinate sequence of a graphic representing the area of ​​the monitoring target, vector data, or the like, is stored. When the monitoring target object itself is designated, for example, information representing the position where the monitoring target object was detected in a predetermined frame constituting a moving image is stored. Also, for example, the state of the monitoring target at the time of processing in step S7 is stored in the storage device 12 as a reference state (in other words, a normal state). The state may be represented, for example, using at least one of the color, brightness, and saturation of pixels in the monitoring target area, or may be represented using the position (for example, coordinates) of the detected object in the image.

[0028] After step S7, or if it is determined in step S6 that a monitoring target has not been input (step S6: NO), processor 11 determines whether to terminate the setting process in Fig. 3 (Fig. 3: step S8). For example, processor 11 determines to terminate the setting process when the user performs an operation to terminate the setting process, or when a confirmation dialog box is displayed after specifying a monitoring target and the user performs an operation to confirm that there are no problems with the specification of the monitoring target.

[0029] If it is determined in step S8 that the setting process is to be ended (step S8: YES), processor 11 ends the process in Fig. 3. On the other hand, if it is determined in step S8 that the setting process is not to be ended (step S8: NO), processor 11 returns to step S1 and repeats the process.

[0030] <Monitoring process> FIG. 7 is a processing flow diagram showing an example of monitoring processing. The in-vehicle device 10 starts the monitoring processing, for example, in response to a user operation. The in-vehicle device 10 may receive information instructing the start of processing from a mobile terminal carried by the user via a network or by ad hoc connection. The camera 20 continuously stores captured image data in the storage device 12 of the in-vehicle device 10. As with the setting processing, an object detection model for detecting objects from image data may be stored in advance in the storage device 12.

[0031] The processor 11 of the in-vehicle device 10 acquires image data output by the camera 20 (FIG. 7: step S11). The processing of this step is the same as step S1 in FIG. 3. After step S11, the processor 11 synthesizes the images (FIG. 7: step S12). The processing of this step is the same as step S2 in FIG. 3.

[0032] After step S12, the processor 11 uses the composite image created in S12 to detect changes from the reference state of the monitored object (FIG. 7: step S13). If the monitored object is set by a range, changes between frames constituting the video can be detected based on the amount of change in hue, etc. within the range. In addition, existing tracking methods can be used to detect changes within the composite image. Moving objects or people may be tracked in the area. Note that if the area to be monitored is an area surrounded by multiple selected objects or a larger area surrounding all of the selected objects, when the selected objects are moved, the area to be monitored may be updated (reset) using the tracked position of the object after the movement. Since the user may use an item even during the monitoring process, updating the area to be monitored allows the monitoring process to be performed without impairing the user's convenience.

[0033] After step S13, processor 11 determines whether an abnormality exists in the monitored object (FIG. 7: step S14). In this step, if the change detected in step S13 exceeds a predetermined tolerance, it is determined that an abnormality exists. If the monitored object is set by a range, it may be determined that an abnormality has occurred, for example, when an object (in other words, an area with a characteristic hue, etc.) that existed within the monitored object area in the reference state moves outside the monitored object area, or when a moving object, such as a person, that was not present within the monitored object area in the reference state enters the monitored object area. Also, if the monitored object area is set using detected objects, it may be determined that an abnormality has occurred, for example, when a selected object moves by more than a threshold in the image or no longer exists within the captured area.

[0034] If it is determined in step S14 that an abnormality exists (step S14: YES), the processor 11 executes processing for abnormality (FIG. 7: step S15). In this step, for example, a security alarm (e.g., a horn, etc.) provided in the vehicle 1 may be sounded to warn the user and third parties (e.g., a person who has entered the area to be monitored). Alternatively, for example, the vehicle 1 may turn on or flash lights such as hazard lights to warn the user and third parties. Note that the sounding of the security alarm or the flashing of lights may be stopped after a predetermined period of time, or may be continued until the user performs an operation to stop the alarm. Also, in step S15, a warning may be issued by sending a message via a network to a mobile device such as a smartphone carried by the user. Alternatively, a video may be stored (recorded) in the storage device 12 without issuing a warning.

[0035] After step S15, processor 11 determines whether to end the monitoring process (FIG. 7: step S16). For example, when an instruction to end the monitoring process is given by a user operation, processor 11 determines to end the monitoring process.

[0036] If it is determined in step S16 that the monitoring process should be terminated (step S16: YES), processor 11 terminates the process in Fig. 7. On the other hand, if it is determined in step S16 that the monitoring process should not be terminated (step S16: NO), or if it is determined in step S14 that there is no abnormality (step S14: NO), processor 11 returns to step S11 and repeats the process.

[0037] <Effects> 3, a monitoring target can be easily set for an image. In particular, if the user can specify the range of the monitoring target for the overhead image, an interface that is intuitively easy for the user to understand can be provided. Furthermore, the overhead image allows the user to easily specify the range including the periphery of the vehicle 1, and the in-vehicle device 10 can monitor the entire periphery of the vehicle 1.

[0038] <Variation 1> For example, a whitelist of people may be registered in advance in the setting process. For example, a face image or a whole-body image of the person is taken by the camera 20 and registered. In this case, in the monitoring process (FIG. 7: step S14), existing face recognition processing or person recognition processing is performed. Even if a person registered on the whitelist is photographed (for example, even if they enter the area being monitored), they will not be judged as abnormal. In this way, the accuracy of abnormality judgment can be improved. Similarly, images of personal belongings can be registered in advance and used as the target for judgment of presence or absence.

[0039] <Variation 2> The vehicle 1 may be equipped with a positioning sensor that outputs its own position information, and the mobile device carried by the user may also be equipped with a positioning sensor that outputs its own position information. During the monitoring process, whether a person captured in a composite image is a suspicious individual may be determined by taking into account the relationship between the position information of the vehicle 1 and the position information of the user's mobile device. The positioning sensor may be, for example, a receiver that receives signals from satellites in a global navigation satellite system (GNSS), calculates coordinates indicating the position of the positioning sensor (i.e., the position of the vehicle 1 or the mobile device), and outputs the coordinates as position information. For example, during the setting process, a whitelist of belongings (mobile devices) carried by the user may be registered. During the monitoring process, the in-vehicle device 10 may communicate with the user's mobile device via a network or an ad hoc connection, and continuously acquire the identification information and position information of the mobile device. Then, in step S14 of FIG. 7, if a person is photographed in the monitored area and the user's mobile device is located in that direction, as determined based on the position information of the mobile device, no abnormality may be detected. This modification also improves the accuracy of abnormality determination.

[0040] <Variation 3> Instead of combining images captured by multiple cameras, the above-described monitoring target may be specified for an image captured by, for example, a single camera. In this case, too, a simple method for setting the monitoring target can be provided.

[0041] <Other> The illustrated system and device configurations are merely examples and are not limited to the above examples. For example, at least a portion of the processing performed by the on-board device 10 of the vehicle 1 may be shared and executed by multiple devices, or may be executed in parallel by multiple devices. Similarly, the processing shown in the processing flow may be executed in a different order or in parallel, as long as the results are unchanged.

[0042] The present invention also includes a computer program for executing the above-described processing method, and a computer-readable recording medium on which the program is recorded. The above-described processing is possible by loading the recording medium into a computer and executing the recorded program. A computer-readable recording medium is a recording medium that stores information such as data and programs through electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer. Among such recording media, those that are removable from a computer include flexible disks, magneto-optical disks, optical disks, magnetic tape, memory cards, etc. Furthermore, recording media that are fixed to a computer include hard disk drives and ROMs, etc. [Explanation of symbols]

[0043] 1: Vehicle, 10: In-vehicle device (monitoring device), 11: Processor, 12: Storage device, 13: User interface (UI), 20: Camera

Claims

1. receiving an operation from a user to designate an area in an image of the surroundings of the vehicle created using a camera mounted on the vehicle; Setting the designated area as a monitoring target; A monitoring device comprising a control unit that executes the above.

2. The operation of specifying the area is an operation in which the user drags on the image via a touch panel stacked on a display that displays the image to specify the periphery of the area. The monitoring device of claim 1 .

3. The operation of specifying the area is an operation of specifying three or more points on the image displayed on the display and thereby specifying an area enclosed by connecting the three or more points. The monitoring device of claim 1 .

4. The control unit further performs object detection to detect an object captured in the image; The operation of specifying the area is an operation of selecting three or more objects from the detected objects and specifying the area surrounded by the three or more objects. The monitoring device of claim 1 .

5. The movement of each of the three or more selected objects is tracked within a moving image formed by the plurality of images, and the monitoring target is updated based on an area surrounded by the three or more objects after the movement. The monitoring device according to claim 4.

6. The control unit The image capturing device further executes the process of creating an overhead image of the vehicle by combining the images captured by the plurality of cameras. An operation to designate the area on the overhead image is accepted from the user. The monitoring device of claim 1 .

7. The control unit further performs creating a whitelist of the person or the person's belongings; If a person is photographed in the image, and it is determined that the photographed person or the person's belongings are not on the whitelist, a warning is output to the person or a notification is given to the user. The monitoring device of claim 1 .

8. A monitoring device according to any one of claims 1 to 7; The camera mounted on the vehicle; A monitoring system comprising:

9. receiving a user's operation to specify a range in an image of the surroundings of the vehicle captured by a camera mounted on the vehicle; Setting the designated range as a monitoring target; The computer performs the monitoring method.

10. The range is calculated for an image of the surroundings of the vehicle taken by a camera mounted on the vehicle. Accepting an operation by a specified user; Setting the designated range as a monitoring target; A monitoring program that causes a computer to run the following.

Images