Imaging apparatus and program

Abstract

To provide an imaging apparatus and a program capable of following a monitoring target selected by a user and reproducing it.SOLUTION: An imaging apparatus 2 comprises a control unit 3 which executes display control of a captured image G obtained by imaging an environment surrounding a vehicle 1. The control unit recognizes a monitoring target K included in a captured image, generates a processed captured image H obtained by adding an emphasis display M enhancing the visibility of the monitoring target to the captured image, displays the processed captured image on a display unit 6, follows the monitoring target based on an input operation performed by a user to select the monitoring target having the emphasis display added thereto, and causes it to be displayed on the display unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device and a program for tracking a monitoring target designated by a user.

Background Art

[0002] Patent Document 1 describes an imaging device including a first camera and a second camera capable of imaging a range of more than a hemisphere, and capable of imaging a range of more than a full sphere. According to the imaging device described in Patent Document 1, an imaging image with reduced dead angles can be obtained.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] If cameras are added to reduce dead angles as in the imaging device described in Patent Document 1, imaging images with different imaging directions also increase according to the number of cameras. Therefore, according to the technique described in Patent Document 1, when reproducing an imaging image in which an object to be monitored by the user is imaged, the user needs to select a specific imaging image from among a plurality of different imaging images, and there is a possibility that the operation and reproduction modes become complicated.

[0005] An object of the present invention is to provide an imaging device and a program capable of tracking and reproducing a monitoring target selected by a user.

Means for Solving the Problems

[0006] One aspect of the present invention includes a control unit that executes display control of a captured image obtained by imaging the environment around a vehicle. The control unit recognizes a monitoring target included in the captured image, generates a processed captured image with an enhanced display that enhances the visibility of the monitoring target added to the captured image, displays the processed captured image on a display unit, and tracks the monitoring target based on an input operation by a user to select the monitoring target with the enhanced display and causes it to be displayed on the display unit, and is an imaging device.

Advantages of the Invention

[0007] According to the present invention, among the captured images captured by the imaging device, it is possible to track and reproduce a monitoring target selected by the user.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Embodiments for Carrying Out the Invention

[0009] As shown in FIGS. 1 and 2, the vehicle system S is composed of a vehicle 1, a terminal device 30 held by a user, and a server device 20 that executes processing related to imaging, which are communicably connected to each other via a network W. The vehicle 1 is provided with an imaging device 2 capable of imaging the environment around the vehicle. The vehicle 1 communicates with the server device 20 and mutually transmits and receives data and programs related to the imaging image captured by the imaging device 2. The vehicle 1 and the server device 20 execute processing related to the imaging image based on an input operation input to the terminal device 30.

[0010] The vehicle 1 is, for example, a four-wheeled automobile provided with the imaging device 2. The vehicle 1 may be a manually driven vehicle or an autonomously driven vehicle. The vehicle 1 may be a two-wheeled vehicle, a three-wheeled vehicle, or an automobile having four or more wheels other than four wheels.

[0011] The imaging device 2 captures imaging images such as moving images and still images. When the vehicle 1 is running, the imaging device 2 functions as a drive recorder. The imaging device 2 includes, for example, a camera 2A that images the environment around the vehicle 1 for each predetermined imaging range. The camera 2A includes a plurality of camera devices that image the periphery of the vehicle 1 for each predetermined range. The imaging range of each camera device is individually set so that there is no dead angle with respect to the environment around the vehicle 1.

[0012] In the illustrated example, the camera 2A is composed of four camera devices. The camera 2A includes, for example, a first camera 2B that images a first imaging range R1. The first camera 2B images a first imaging range R1 having, for example, a predetermined angular range in the traveling direction of the vehicle 1 (in FIG. 2: +Y direction) as an angular field of view.

[0013] The camera 2A includes, for example, a second camera 2C that images a second imaging range R2. The second camera 2C images a second imaging range R2 having, for example, a predetermined angular range in the left direction (in FIG. 2: -X direction) as viewed in the traveling direction of the vehicle 1 as an angular field of view. There is an overlapping angular range between the second imaging range R2 and the first imaging range R1.

[0014] The camera 2A includes, for example, a third camera 2D that images a third imaging range R3. The third camera 2D images, for example, a third imaging range R3 having a predetermined angular range that faces the right side (in the +X direction in FIG. 2) when viewed in the traveling direction of the vehicle 1. There is an overlapping angular range between the third imaging range R3 and the first imaging range R1.

[0015] The camera 2A includes, for example, a fourth camera 2E that images a fourth imaging range R4. The fourth camera 2E images, for example, a fourth imaging range R4 having a predetermined angular range that faces the opposite direction (in the -Y direction in FIG. 2) with respect to the traveling direction of the vehicle 1. There is an overlapping angular range between the fourth imaging range R4 and the second imaging range R2. There is an overlapping angular range between the fourth imaging range R4 and the third imaging range R3.

[0016] The camera 2A may be composed not only of a plurality of camera devices but also of one omnidirectional camera having a 360-degree imaging range or two hemispherical cameras having an 180-degree imaging range. The camera 2A may be composed of any camera device as long as it can reduce blind spots.

[0017] The first camera 2B, the second camera 2C, the third camera 2D, and the fourth camera 2E are controlled by a control unit 3 provided in the imaging device 2. The control unit 3 reads data and programs necessary for control from a storage unit 4 provided in the imaging device 2 and controls the camera 2A. The control unit 3 executes display control of the captured images captured by the camera 2A. The captured images are stored in the storage unit 4 based on a predetermined file format at predetermined time intervals.

[0018] The storage unit 4 stores the captured images captured by the first camera 2B, the second camera 2C, the third camera 2D, and the fourth camera 2E for a predetermined time. When there is no event in the environment around the vehicle 1, the captured images are discarded from the storage unit 4 at a predetermined timing in the order of oldest, and are always stored in the storage unit 4 in the order of newest. When there is an event in the environment around the vehicle 1, the captured images are stored in the storage unit 4 going back a predetermined time.

[0019] The control unit 3 is composed of a hardware processor such as at least one CPU (Central Processing Unit). The storage unit 4 is composed of a non-volatile storage medium such as a hard disk drive (HDD) or a solid state drive (SSD).

[0020] The vehicle 1 is provided with an operation unit 5 for receiving a user's input operation on the imaging device 2. The operation unit 5 is composed of an input interface such as a physical button or a jog dial. The user operates the operation unit 5 to operate the imaging device 2. The vehicle 1 is provided with a display unit 6 for displaying the captured image captured by the imaging device 2.

[0021] The display unit 6 is composed of an image-displayable display device such as a liquid crystal display. When the display unit 6 is composed of a touch panel, it may be integrated with the operation unit 5 by displaying an input display image for receiving a user's input operation. The operation unit 5 and the display unit 6 may be composed of a terminal device 30 held by the user.

[0022] The vehicle 1 is provided with a notification unit 7 for outputting a predetermined notification inside or outside the vehicle. The notification unit 7 is composed of a speaker for voice output, an electroluminescent display device for outputting characters or images, a light-emitting device for outputting light based on a predetermined output pattern, etc. The vehicle 1 is provided with a communication unit 8 capable of communicating with the network W. The communication unit 8 is a communication interface for communicating with the server device 20 and the terminal device 30 via the network W. The communication unit 8 includes, for example, a communication interface capable of mobile communication, a wireless interface such as wi-fi (registered trademark) or Bluetooth (registered trademark).

[0023] The server device 20 communicates with the vehicle 1 via the network W and acquires captured images. The server device 20 includes an arithmetic unit 21 that executes predetermined processing on the captured images. The server device 20 includes a storage unit 22 that stores data and programs necessary for the processing of the arithmetic unit 21. The server device 20 includes a communication unit 23 configured by a communication interface capable of communicating via the network W. The arithmetic unit 21 is configured by a hardware processor such as at least one CPU.

[0024] The storage unit 22 is configured by a non-temporary storage medium such as a hard disk drive or a solid state disk. The communication unit 23 is configured by, for example, a wireless interface or a wired interface connectable to the network W.

[0025] The terminal device 30 is a wirelessly communicable information processing terminal device such as a smartphone, a personal computer, or a tablet terminal device. The terminal device 30 receives an input operation regarding the imaging device 2 by communicating via the network W or directly with the vehicle 1. The terminal device 30 acquires and displays a captured image by communicating with the vehicle 1.

[0026] The terminal device 30 includes a control unit 31 that executes processing necessary for the input operation and the display of the captured image. The terminal device 30 includes a storage unit 32 that stores data and programs necessary for the processing of the control unit 31. The control unit 31 is configured by a hardware processor such as at least one CPU. The storage unit 32 is configured by a non-temporary storage medium such as a hard disk drive or a solid state disk.

[0027] The terminal device 30 includes a display unit 33 that receives an input operation and displays a display image. The display unit 33 is configured by, for example, a touch panel type liquid crystal display. The communication unit 34 includes, for example, a communication interface capable of mobile communication, a wireless interface such as wi-fi (registered trademark), or Bluetooth (registered trademark).

[0028] With the above configuration, the imaging device 2 detects a monitoring target that may affect the vehicle 1 based on the captured image of the environment around the vehicle 1. The monitoring target includes, for example, moving objects such as a person, an animal, a vehicle, a drone, etc. approaching within a predetermined distance range of the vehicle 1. The imaging device 2 operates as a monitoring camera that captures the environment around the vehicle 1 when the vehicle 1 is stopped, for example.

[0029] As shown in FIG. 3, when the vehicle 1 is parked, the control unit 3 acquires the captured image G captured by the imaging device 2. The captured image G includes a plurality of divided captured images captured by the imaging device 2 for each imaging range. The control unit 3 acquires, for example, the captured image G including the first divided captured image G1 captured by the first camera 2B, the second divided captured image G2 captured by the second camera 2C, the third divided captured image G3 captured by the third camera 2D, and the fourth divided captured image G4 captured by the fourth camera 2E. The control unit 3 generates the captured image G by synthesizing the plurality of divided captured images.

[0030] The control unit 3 recognizes the monitoring target K included in the captured image G. Here, the control unit 3 is set to be able to extract the monitoring target K included in the captured image G by performing machine learning using deep learning based on teacher data in advance. When the control unit 3 recognizes the monitoring target K, it determines that an event has occurred in the environment around the vehicle 1, stores the captured image in the storage unit 4 from the imaging start time a predetermined time back, and prevents the deletion of this captured image.

[0031] The control unit 3 continues to store the captured image in the storage unit 4 until the monitoring target K is no longer recognized. The control unit 3 extracts the monitoring target K, generates a processed captured image (H) with an emphasis display M that enhances the visibility of the monitoring target K added to the captured image G, and stores it in the storage unit 4.

[0032] In the example shown in FIG. 3(A), in the third divided captured image G3 that captures the third imaging range R3 on the right side of the vehicle 1, the monitoring target K appears. The control unit 3 extracts the monitoring target K that appears in the third divided captured image G3 and generates a third processed divided captured image (H3) with an added highlighting display M. In the illustrated example, the highlighting display M is generated in a frame shape surrounding the monitoring target K. If the highlighting display M can enhance the visibility of the monitoring target K, it may be generated by appropriately combining not only the frame display but also colors, arrows, character information, line drawings, blinking displays, etc.

[0033] The control unit 3 moves the highlighting display M according to the movement of the monitoring target K. When the monitoring target moves from one divided captured image to another while the divided captured image is being displayed on the display unit, the control unit 3 generates another divided captured image with the added highlighting display M and stores it in the storage unit 4. When the monitoring target K moves from one divided captured image to another, the control unit 3 generates another divided captured image with the added highlighting display M. In the example shown in FIG. 3(B), the control unit 3 extracts the monitoring target K that appears in the first divided captured image G1 from the third divided captured image G3 and generates a first processed divided captured image (H1) with the added highlighting display M.

[0034] In the example shown in FIG. 3(C), the control unit 3 extracts the monitoring target K that appears in the second divided captured image G2 from the first divided captured image G1 and generates a second processed divided captured image (H2) with the added highlighting display M. In the example shown in FIG. 3(D), the control unit 3 extracts the monitoring target K that appears in the second divided captured image G2 from the first divided captured image G1 and generates a second processed divided captured image (H2) with the added highlighting display M.

[0035] As shown in FIG. 4, the control unit 3 causes the display unit 6 to display the captured image G based on the input operation of the user who displays the captured image G. When the monitored object K is included in the captured image G, the control unit 3 causes the display unit 6 to display a processed captured image H including the monitored object K with the highlight display M added (see FIG. 4(A)). When the user checks the processed captured image H and continues to display the monitored object K, the user performs an input operation to select the monitored object K. The user performs an input operation to select the monitored object K included in the processed captured image H by operating the operation unit 5. The user may also perform an input operation to select the monitored object K included in the processed captured image H by touching the display unit 6 configured by a touch panel.

[0036] Based on the input operation, the control unit 3 selects one of the plurality of divided captured images in which the monitored object K is captured, and causes the display unit 6 to display the selected divided captured image. Based on the input operation, the control unit 3 selects a processed divided captured image in which the highlight display M is added to one of the plurality of divided captured images in which the monitored object K is captured, and causes the display unit 6 to display the processed divided captured image. The control unit 3 enlarges the processed divided captured image or displays it on the display unit 6 in full screen.

[0037] Based on the input operation of the user, the control unit 3 tracks the monitored object K, enlarges the processed divided captured image with the highlight display M added, and causes the display unit 6 to display it. In the example of FIG. 4, the control unit 3 selects the third processed divided captured image H3 from the processed captured image H based on the input operation (see FIG. 4(A)). When the monitored object K moves, the control unit 3 tracks the monitored object K and causes the display unit 6 to display the processed divided captured image (see FIGS. 4(B) to 4(D)).

[0038] The control unit 3 may calculate physical feature amounts, motion feature amounts, and action pattern feature amounts such as position and time zone for individually identifying the monitored object K selected by the user, and store them in the storage unit 4. When the control unit 3 recognizes that the monitored object K has affected the vehicle 1, the control unit 3 may generate a predetermined notification, output it to the notification unit 7, and transmit the predetermined notification to the terminal device 30.

[0039] When a monitoring target K that was previously tracked based on a past input operation is captured again in the captured image G, the control unit 3 may automatically track the monitoring target K based on each feature amount and display it on the display unit 6. The control unit 3 may also display information that the monitoring target K was recognized in the past on the display unit 6. When a monitoring target K that was previously tracked is captured again in the captured image G, the control unit 3 may generate a predetermined notification and transmit it to the terminal device 30.

[0040] When the user confirms the notification, the user may cause the notification unit 7 of the vehicle 1 to output a predetermined notification by performing an input operation on the terminal device 30. When a monitoring target K that was previously tracked is captured again in the captured image G, the control unit 3 may cause the notification unit 7 to automatically output a predetermined notification.

[0041] The control unit 3 may transmit the calculated feature amount of the monitoring target K to the server device 20. The server device 20 may classify the monitoring target K based on data aggregated from a plurality of vehicles 1, and provide information to a plurality of vehicles 1 and a plurality of terminal devices 30. The control unit 3 may automatically recognize the monitoring target K based on the feature amount of the information provided, automatically generate and store a processed captured image, and transmit a predetermined notification to the terminal device 30 and the server device 20. The control unit 3 may generate the processed captured image by adding text information related to the monitoring target K. When the monitoring target K is notified from the vehicle 1, the server device 20 may notify a predetermined notification to the vehicle 1 present within a predetermined range from the current position of the monitoring target K and the terminal device 30 owned by the user of the vehicle 1.

[0042] The server device 20 generates feature quantities of the monitoring target K based on information on the monitoring target K provided by security authorities such as the police and information on the monitoring target K present on the network W, and when the monitoring target K is notified from the vehicle 1, may issue a predetermined notification to the vehicle 1 present within a predetermined range from the current location of the monitoring target K or to a terminal device 30 owned by the user of the vehicle 1. When the user confirms the notification, he or she may notify the security authorities by performing an input operation on the terminal device 30.

[0043] When the server device 20 is notified of the monitoring target K from the vehicle 1, it may automatically notify the public security authorities of a predetermined notification including the position information of the vehicle 1. Based on the notifications received in real time, the public security authorities may dispatch resources for implementing countermeasures to an area including the position of the vehicle 1 where the monitoring target K is recognized, according to the content of the actions of the monitoring target K.

[0044] Based on the aggregated data of the monitoring target K, the server device 20 analyzes the action pattern of the monitoring target K, calculates the estimated area where the monitoring target K appears and disappears and the time zone thereof, and may transmit a predetermined notification to the terminal device 30 owned by the user of the vehicle 1 included in the estimated area and to the public security authorities. Each process in the above-described control unit 3 may be executed in cooperation with the arithmetic unit 21 of the server device 20 and the control unit 31 of the terminal device 30.

[0045] FIG. 5 shows the processing flow of the imaging method executed in the imaging device 2. The imaging method is executed based on a computer program installed in a computer mounted on the imaging device 2. The control unit 3 acquires an imaging image G obtained by imaging the environment around the vehicle 1 with the camera 2A (step S100). The control unit 3 determines whether or not the monitoring target K is included in the imaging image G (step S102). When the monitoring target K is included in the imaging image G, the control unit 3 generates a processed imaging image H with an added highlighting M and stores it in the storage unit 4 (step S104).

[0046] The control unit 3 starts display control based on a user operation (step S106). The control unit 3 causes the display unit 6 to display the processed imaging image H (step S108). The control unit 3 determines whether or not an input operation for the user to select the monitoring target K with the added highlighting M has been performed (step S110). When an input operation for the user to select the monitoring target K with the added highlighting M has been performed, the control unit 3 tracks the monitoring target K and causes it to be displayed on the display unit 6 (step S112).

[0047] When there is no monitoring target K in step S102, the control unit 3 causes the storage unit 4 to store the captured image G (step S114). The control unit 3 starts display control based on a user operation (step S116). The control unit 3 causes the display unit 6 to display the captured image G (step S118). When no input operation is performed by the user to select the monitoring target K with the highlighting M added in step S110, the control unit 3 causes the display unit 6 to display the captured image G (step S118).

[0048] As described above, according to the imaging device 2, when the monitoring target K is recognized in the captured image G, the processed captured image H with the highlighting M added to the monitoring target K can be generated. According to the imaging device 2, when the processed captured image H is displayed on the display unit 6 and the user performs an input operation to select the monitoring target K, the processed captured image obtained by tracking the monitoring target K can be displayed on the display unit 6.

[0049] In the above-described embodiment, the computer program executed in each component of the vehicle system S may be provided in a form recorded on a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium.

Explanation of Reference Numerals

[0050] 1 Vehicle, 2 Imaging Device, 2A Camera, 3 Control Unit, 4 Storage Unit, 5 Operation Unit, 6 Display Unit, 7 Notification Unit, 8 Communication Unit, 20 Server Device, 21 Arithmetic Unit, 22 Storage Unit, 23 Communication Unit, 30 Terminal Device, 31 Control Unit, 32 Storage Unit, 33 Display Unit, 34 Communication Unit, G Captured Image, G1 First Divided Captured Image, G2 Second Divided Captured Image, G3 Third Divided Captured Image, G4 Fourth Divided Captured Image, H Processed Captured Image, H1 First Processed Divided Captured Image, H2 Second Processed Divided Captured Image, H3 Third Processed Divided Captured Image, K Monitoring Target, M Highlighting, R1 First Imaging Range, R2 Second Imaging Range, R3 Third Imaging Range, R4 Fourth Imaging Range, S Vehicle System, W Network

Claims

1. An imaging device comprising a control unit that executes display control of an imaging image obtained by imaging the environment around a vehicle, wherein the control unit recognizes a monitoring target included in the imaging image, generates a processed imaging image with an emphasis display added to enhance the visibility of the monitoring target in the imaging image, displays the processed imaging image on a display unit, and tracks the monitoring target and causes it to be displayed on the display unit based on an input operation in which a user selects the monitoring target with the emphasis display added thereto. Imaging device.

2. The control unit generates a processed divided imaging image with the emphasis display added to one of the plurality of divided imaging images obtained by dividing the environment into predetermined ranges and imaging each range, in which the monitoring target is imaged, selects the processed divided imaging image based on the input operation, and causes the processed divided imaging image to be displayed on the display unit. The imaging device according to claim 1.

3. The control unit while the divided imaging image is being displayed on the display unit, causes another divided imaging image to be displayed on the display unit when the monitoring target moves from one divided imaging image to another divided imaging image. The imaging device according to claim 2.

4. The control unit automatically tracks the monitoring target and causes it to be displayed on the display unit when the monitoring target tracked in the past is imaged again in the imaging image. The imaging device according to claim 2.

5. A program for causing a computer to execute a process of obtaining an imaging image obtained by imaging the environment around a vehicle, recognizing a monitoring target included in the imaging image, generating a processed imaging image with an emphasis display added to enhance the visibility of the monitoring target in the imaging image, displaying the processed imaging image on a display unit, and tracking the monitoring target and causing it to be displayed on the display unit based on an input operation in which a user selects the monitoring target with the emphasis display added thereto. Program.

Images