Imaging device and program

Abstract

To provide an imaging apparatus and a program capable of following a monitoring target selected by a user and reproducing it.SOLUTION: An imaging apparatus 2 comprises a control unit 3 which executes display control of a captured image G obtained by imaging an environment surrounding a vehicle 1. The control unit recognizes a monitoring target K included in a captured image, generates a processed captured image H obtained by adding an emphasis display M enhancing the visibility of the monitoring target to the captured image, displays the processed captured image on a display unit 6, follows the monitoring target based on an input operation performed by a user to select the monitoring target having the emphasis display added thereto, and causes it to be displayed on the display unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device and a program for tracking a monitoring target specified by a user.

Background Art

[0002] Patent Document 1 describes an imaging device including a first camera and a second camera capable of imaging a range of more than a hemisphere, and capable of imaging a range of more than a full sphere. According to the imaging device described in Patent Document 1, an imaging image with reduced dead angles can be obtained.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When additional cameras are installed to reduce dead angles as in the imaging device described in Patent Document 1, the number of imaging images with different imaging directions also increases according to the number of cameras. Therefore, according to the technology described in Patent Document 1, when reproducing an imaging image in which an object to be monitored by the user is imaged, the user needs to select a specific imaging image from a plurality of different imaging images, and the operation and reproduction modes may become complicated.

[0005] An object of the present invention is to provide an imaging device and a program capable of tracking and reproducing a monitoring target selected by a user.

Means for Solving the Problems

[0006] One aspect of the present invention is an imaging device comprising a control unit that performs display control of an image captured of the environment surrounding a vehicle, the control unit recognizes a target to be monitored included in the image, generates a processed image by adding an emphasis to the image to enhance the visibility of the target to be monitored, displays the processed image on a display unit, tracks the target to be monitored based on an input operation in which the user selects the target to which the emphasis has been added, and displays the target on the display unit. [Effects of the Invention]

[0007] According to the present invention, the imaging device can track and play back images of a target being monitored, selected by the user, from among the captured images. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the configuration of a vehicle system according to the embodiment. [Figure 2] This diagram shows the imaging range of the imaging device installed on the vehicle. [Figure 3] This figure shows an example of a processed image in which the monitored target included in the captured image has been highlighted. [Figure 4] This figure shows an example of a state where processed images of a tracked target are displayed. [Figure 5] This is a flowchart showing the processing flow of the imaging method performed in the imaging device. [Modes for carrying out the invention]

[0009] As shown in Figures 1 and 2, the vehicle system S consists of a vehicle 1, a terminal device 30 owned by the user, and a server device 20 that performs imaging-related processing, all of which are connected to each other via a network W. The vehicle 1 is equipped with an imaging device 2 capable of capturing images of the environment around the vehicle. The vehicle 1 communicates with the server device 20 and mutually sends and receives data and programs related to the images captured by the imaging device 2. The vehicle 1 and the server device 20 perform processing related to the captured images based on input operations input to the terminal device 30.

[0010] Vehicle 1 is, for example, a four-wheeled automobile equipped with an imaging device 2. Vehicle 1 may be a manually driven vehicle or an automatically driven vehicle. Vehicle 1 may also be a two-wheeled vehicle, a three-wheeled vehicle, or an automobile with four or more wheels.

[0011] The imaging device 2 captures images such as videos and still images. When the vehicle 1 is in motion, the imaging device 2 functions as a drive recorder. The imaging device 2 includes, for example, a camera 2A that captures the environment around the vehicle 1 at predetermined imaging ranges. Camera 2A comprises multiple camera devices that capture the area around the vehicle 1 at predetermined ranges. Each camera device has its imaging range individually set so that there are no blind spots in the environment around the vehicle 1.

[0012] In the illustrated example, camera 2A is composed of four camera devices. Camera 2A includes, for example, a first camera 2B that captures a first imaging range R1. The first camera 2B captures a first imaging range R1 whose field of view is a predetermined angular range in the direction of travel of the vehicle 1 (Figure 2: +Y direction).

[0013] Camera 2A includes, for example, a second camera 2C that captures a second imaging range R2. The second camera 2C captures a second imaging range R2 whose field of view is a predetermined angular range directed to the left (Figure 2: -X direction) when viewed in the direction of travel of the vehicle 1. There is an overlapping angular range between the second imaging range R2 and the first imaging range R1.

[0014] Camera 2A includes, for example, a third camera 2D that captures a third imaging range R3. The third camera 2D captures a third imaging range R3 whose field of view is a predetermined angular range directed to the right (Figure 2: +X direction) when viewed in the direction of travel of vehicle 1. There is an overlapping angular range between the third imaging range R3 and the first imaging range R1.

[0015] Camera 2A includes, for example, a fourth camera 2E that images the fourth imaging range R4. The fourth camera 2E images the fourth imaging range R4, which has a predetermined angular field of view that is in the opposite direction to the direction of travel of the vehicle 1 (Figure 2: -Y direction). There is an overlapping angular range between the fourth imaging range R4 and the second imaging range R2. There is also an overlapping angular range between the fourth imaging range R4 and the third imaging range R3.

[0016] Camera 2A may consist of multiple camera devices, as well as one 360-degree spherical camera with a 360-degree imaging range and two hemispherical cameras with a 180-degree imaging range. Camera 2A may consist of any camera device that can reduce blind spots.

[0017] The first camera 2B, the second camera 2C, the third camera 2D, and the fourth camera 2E are controlled by a control unit 3 provided in the imaging device 2. The control unit 3 reads the data and programs necessary for control from the storage unit 4 provided in the imaging device 2 and controls camera 2A. The control unit 3 performs display control of the captured images captured by camera 2A. The captured images are stored in the storage unit 4 at predetermined time intervals based on a predetermined file format.

[0018] The storage unit 4 stores the images captured by the first camera 2B, the second camera 2C, the third camera 2D, and the fourth camera 2E for a predetermined period of time. If there are no events in the environment surrounding the vehicle 1, the captured images are discarded from the storage unit 4 in chronological order at a predetermined time, and are always stored in the storage unit 4 in chronological order. If there are events in the environment surrounding the vehicle 1, the captured images are stored in the storage unit 4 retrospectively for a predetermined period of time.

[0019] The control unit 3 is composed of a hardware processor such as at least one CPU (Central Processing Unit). The storage unit 4 is composed of a non-temporary storage medium such as a hard disk drive (HDD) or a solid state disk (SSD).

[0020] The vehicle 1 is provided with an operation unit 5 for receiving a user's input operation on the imaging device 2. The operation unit 5 is composed of an input interface such as a physical button or a jog dial. The user operates the operation unit 5 to operate the imaging device 2. The vehicle 1 is provided with a display unit 6 for displaying the captured image captured by the imaging device 2.

[0021] The display unit 6 is composed of an image displayable display device such as a liquid crystal display. When the display unit 6 is composed of a touch panel, it may be integrated with the operation unit 5 by displaying an input display image for receiving a user's input operation. The operation unit 5 and the display unit 6 may be composed of a terminal device 30 possessed by the user.

[0022] The vehicle 1 is provided with a notification unit 7 for outputting a predetermined notification inside or outside the vehicle. The notification unit 7 is composed of a speaker for voice output, an electro-optical display device for outputting characters or images, a light emitting device for outputting light based on a predetermined output pattern, etc. The vehicle 1 is provided with a communication unit 8 capable of communicating with the network W. The communication unit 8 is a communication interface for communicating with the server device 20 and the terminal device 30 via the network W. The communication unit 8 includes, for example, a communication interface capable of mobile communication, a wireless interface such as wi-fi (registered trademark) or Bluetooth (registered trademark).

[0023] The server device 20 communicates with the vehicle 1 via the network W and acquires captured images. The server device 20 includes a processing unit 21 that performs predetermined processing on the captured images. The server device 20 includes a storage unit 22 that stores data and programs necessary for the processing of the processing unit 21. The server device 20 includes a communication unit 23 configured with a communication interface that can communicate via the network W. The processing unit 21 is configured with at least one hardware processor such as a CPU.

[0024] The storage unit 22 is composed of non-temporary storage media such as a hard disk drive or a solid-state disk. The communication unit 23 is composed of, for example, a wireless interface or a wired interface that can connect to the network W.

[0025] The terminal device 30 is a wireless communication-enabled information processing terminal device, such as a smartphone, personal computer, or tablet terminal device. The terminal device 30 receives input operations related to the imaging device 2 via the network W or by communicating directly with the vehicle 1. The terminal device 30 acquires and displays the captured image by communicating with the vehicle 1.

[0026] The terminal device 30 includes a control unit 31 that performs processing necessary for input operations and displaying captured images. The terminal device 30 also includes a storage unit 32 that stores data and programs necessary for the processing of the control unit 31. The control unit 31 is composed of at least one hardware processor such as a CPU. The storage unit 32 is composed of a non-temporary storage medium such as a hard disk drive or a solid-state disk.

[0027] The terminal device 30 includes a display unit 33 that accepts input operations and displays images. The display unit 33 is composed of, for example, a touch panel type liquid crystal display. The communication unit 34 includes, for example, a mobile communication interface, a wireless interface such as Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0028] With the above configuration, the imaging device 2 detects potential monitoring targets that may affect the vehicle 1 based on captured images of the environment surrounding the vehicle 1. Monitoring targets include, for example, people, animals, vehicles, drones, and other moving objects that approach within a predetermined distance range of the vehicle 1. When the vehicle 1 is stationary, the imaging device 2 operates as a surveillance camera that captures images of the environment surrounding the vehicle 1.

[0029] As shown in Figure 3, when the vehicle 1 is parked, the control unit 3 acquires the captured image G taken by the imaging device 2. The captured image G includes multiple segmented captured images taken by the imaging device 2 for each imaging range. For example, the control unit 3 acquires the captured image G which includes the first segmented captured image G1 taken by the first camera 2B, the second segmented captured image G2 taken by the second camera 2C, the third segmented captured image G3 taken by the third camera 2D, and the fourth segmented captured image G4 taken by the fourth camera 2E. The control unit 3 synthesizes the multiple segmented captured images to generate the captured image G.

[0030] The control unit 3 recognizes the target K to be monitored, which is included in the captured image G. Here, the control unit 3 is configured to extract the target K to be monitored, which is included in the captured image G, by performing machine learning using deep learning based on training data in advance. When the control unit 3 recognizes the target K to be monitored, it determines that an event has occurred in the environment surrounding the vehicle 1, and stores the captured image in the storage unit 4 from the start time of imaging, which is set back a predetermined time, to prevent the deletion of this captured image.

[0031] The control unit 3 continues to store the captured image in the storage unit 4 until the monitored object K is no longer recognized. The control unit 3 extracts the monitored object K, generates a processed captured image (H) by adding an emphasis M to the captured image G to enhance the visibility of the monitored object K, and stores it in the storage unit 4.

[0032] In the example shown in Figure 3(A), the monitored object K appears in the third segmented image G3, which captures the third imaging range R3 on the right side of vehicle 1. The control unit 3 extracts the monitored object K that appeared in the third segmented image G3 and generates a third processed segmented image (H3) with a highlighting M added. In the illustrated example, the highlighting M is generated in the shape of a frame surrounding the monitored object K. The highlighting M may be generated by combining colors, arrows, text information, line drawings, flashing displays, etc., as appropriate, rather than just a frame, as long as it enhances the visibility of the monitored object K.

[0033] The control unit 3 moves the highlighting M in accordance with the movement of the monitored object K. While the control unit 3 is displaying the segmented captured images on the display unit, if the monitored object moves from one segmented captured image to another segmented captured image, the control unit 3 generates another segmented captured image with the highlighting M added and stores it in the storage unit 4. The control unit 3 generates another segmented captured image with the highlighting M added when the monitored object K moves from one segmented captured image to another segmented captured image. In the example shown in Figure 3(B), the control unit 3 extracts the monitored object K that appeared in the first segmented captured image G1 from the third segmented captured image G3 and generates a first processed segmented captured image (H1) with the highlighting M added.

[0034] In the example shown in Figure 3(C), the control unit 3 extracts the monitored object K that appeared in the second divided image G2 from the first divided image G1 and generates a second processed divided image (H2) with added highlighting M. In the example shown in Figure 3(D), the control unit 3 extracts the monitored object K that appeared in the second divided image G2 from the first divided image G1 and generates a second processed divided image (H2) with added highlighting M.

[0035] As shown in Figure 4, the control unit 3 displays the captured image G on the display unit 6 based on the user's input operation to display the captured image G. If the captured image G contains a target K, the control unit 3 displays the processed captured image H, which includes the target K with the added highlighting M, on the display unit 6 (see Figure 4(A)). The user confirms the processed captured image H and, if they wish to continue displaying the target K, performs an input operation to select the target K. The user performs an input operation to select the target K included in the processed captured image H by operating the operation unit 5. The user may also perform an input operation to select the target K included in the processed captured image H by touching the display unit 6, which is configured as a touch panel.

[0036] Based on the input operation, the control unit 3 selects one segmented image from among multiple segmented images in which the monitored target K is captured, and displays the selected segmented image on the display unit 6. Based on the input operation, the control unit 3 selects a processed segmented image from among multiple segmented images in which the monitored target K is captured, with highlighting M added, and displays the processed segmented image on the display unit 6. The control unit 3 enlarges or displays the processed segmented image in full screen on the display unit 6.

[0037] Based on user input, the control unit 3 tracks the monitored object K and displays the processed segmented image with added highlighting M on the display unit 6. In the example in Figure 4, the control unit 3 selects the third processed segmented image H3 from the processed image H based on the input (see Figure 4(A)). If the monitored object K moves, the control unit 3 tracks the monitored object K and displays the processed segmented image on the display unit 6 (see Figures 4(B) to 4(D)).

[0038] The control unit 3 may calculate physical characteristics, movement characteristics, and behavioral pattern characteristics such as location and time of day to individually identify the monitored target K selected by the user, and store them in the memory unit 4. When the control unit 3 recognizes that the monitored target K has affected the vehicle 1, it may generate a predetermined notification and output it to the notification unit 7, and may also transmit the predetermined notification to the terminal device 30.

[0039] The control unit 3 may, based on each feature quantity, automatically track the monitored target K that was previously tracked based on past input operations and display it on the display unit 6 if it is captured again in the captured image G. The control unit 3 may also display on the display unit 6 information indicating that the monitored target K was previously recognized. When the captured image G is being captured, the control unit 3 may generate a predetermined notification and transmit it to the terminal device 30 if it is captured again in the captured image G if it is a monitored target K that was previously tracked.

[0040] If the user confirms the notification, they may cause the notification unit 7 of the vehicle 1 to output a predetermined notification by performing an input operation on the terminal device 30. The control unit 3 may automatically output a predetermined notification to the notification unit 7 if the monitored target K that was previously tracked is captured in the captured image G again.

[0041] The control unit 3 may transmit the calculated feature quantities of the monitored target K to the server device 20. The server device 20 may classify the monitored target K based on the aggregated data from multiple vehicles 1 and provide information to multiple vehicles 1 and multiple terminal devices 30. Based on the feature quantities received, the control unit 3 may automatically recognize the monitored target K, automatically generate and store a processed image, and transmit a predetermined notification to the terminal devices 30 and the server device 20. The control unit 3 may also generate the processed image by adding text information about the monitored target K. When the monitored target K is notified from a vehicle 1, the server device 20 may notify a predetermined notification to vehicles 1 located within a predetermined range from the current location of the monitored target K, or to terminal devices 30 owned by the users of vehicles 1.

[0042] The server device 20 generates feature quantities for the monitored target K based on information about the monitored target K provided by law enforcement authorities such as the police, and information about the monitored target K present on the network W. If the monitored target K is reported from vehicle 1, the server device 20 may report a predetermined notification to vehicle 1 or terminal devices 30 owned by the user of vehicle 1 that are within a predetermined range from the current location of the monitored target K. If the user confirms the notification, they may notify the law enforcement authorities by performing an input operation on the terminal device 30.

[0043] The server device 20 may automatically notify the security authorities of a predetermined notification, including the location information of vehicle 1, when the monitored target K is notified from vehicle 1. Based on the notification received in real time, the security authorities may dispatch resources to the area including the location of vehicle 1 where the monitored target K was identified, in order to implement countermeasures according to the nature of the monitored target K's actions.

[0044] The server device 20 may analyze the behavioral patterns of the monitored target K based on the aggregated data of the monitored target K, calculate the estimated area and time period in which the monitored target K is likely to appear, and send a predetermined notification to the terminal device 30 owned by the user of the vehicle 1 included in the estimated area, or to the security authorities. Each of the processes in the control unit 3 described above may be executed in cooperation with the calculation unit 21 of the server device 20 and the control unit 31 of the terminal device 30.

[0045] Figure 5 shows the processing flow of the imaging method performed in the imaging device 2. The imaging method is performed based on a computer program installed in the computer mounted on the imaging device 2. The control unit 3 acquires an image G of the environment surrounding the vehicle 1 captured by the camera 2A (step S100). The control unit 3 determines whether or not the image G contains the target of monitoring K (step S102). If the image G contains the target of monitoring K, the control unit 3 generates a processed image H with added highlighting M and stores it in the storage unit 4 (step S104).

[0046] The control unit 3 starts display control based on the user's operation (step S106). The control unit 3 displays the processed image H on the display unit 6 (step S108). The control unit 3 determines whether or not the user has made an input operation to select the monitored object K to which the highlighting M has been added (step S110). If the user has made an input operation to select the monitored object K to which the highlighting M has been added, the control unit 3 tracks the monitored object K and displays it on the display unit 6 (step S112).

[0047] If no monitoring target K exists in step S102, the control unit 3 stores the captured image G in the storage unit 4 (step S114). The control unit 3 starts display control based on user operation (step S116). The control unit 3 displays the captured image G on the display unit 6 (step S118). If the user does not input an operation to select a monitoring target K to which highlighting M has been added in step S110, the control unit 3 displays the captured image G on the display unit 6 (step S118).

[0048] As described above, according to the imaging device 2, when a target K is recognized in the captured image G, a processed captured image H can be generated by adding a highlight M to the target K. According to the imaging device 2, when the processed captured image H is displayed on the display unit 6, if the user performs an input operation to select a target K, the processed captured image tracking the target K can be displayed on the display unit 6.

[0049] In the embodiments described above, the computer programs executed in each configuration of the vehicle system S may be provided in the form of being recorded on a computer-readable portable recording medium, such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. [Explanation of symbols]

[0050] 1 Vehicle, 2 Imaging device, 2A Camera, 3 Control unit, 4 Storage unit, 5 Operation unit, 6 Display unit, 7 Notification unit, 8 Communication unit, 20 Server device, 21 Calculation unit, 22 Storage unit, 23 Communication unit, 30 Terminal device, 31 Control unit, 32 Storage unit, 33 Display unit, 34 Communication unit, G Captured image, G1 First segmented captured image, G2 Second segmented captured image, G3 Third segmented captured image, G4 Fourth segmented captured image, H Processed captured image, H1 First processed segmented captured image, H2 Second processed segmented captured image, H3 Third processed segmented captured image, K Monitoring target, M Highlighting, R1 First imaging range, R2 Second imaging range, R3 Third imaging range, R4 Fourth imaging range, S Vehicle system, W Network

Claims

1. It includes a control unit that performs display control of captured images of the environment surrounding the vehicle, The control unit, The target to be monitored is recognized in the aforementioned captured image, A processed image is generated by adding an enhanced display to the aforementioned captured image to improve the visibility of the monitored target. The processed image is displayed on the display unit. Based on the user's input operation to select the monitored target to which the aforementioned highlighting has been added, the monitored target is tracked and displayed on the display unit. The control unit, From among the multiple segmented images obtained by dividing the environment into predetermined ranges and capturing them, a processed segmented image is generated by adding the highlighting to one of the segmented images in which the monitored target is captured. Based on the input operation, select the processed segmented image and display the processed segmented image on the display unit. The control unit, While the segmented image is being displayed on the display unit, When the monitored object moves from one of the segmented image captures to another of the segmented image captures, the other segmented image captures is displayed on the display unit. The control unit, An imaging device that automatically tracks the monitored target and displays it on the display unit if the monitored target that was previously tracked is captured in the captured image again.

2. Acquire an image of the environment surrounding the vehicle, The target to be monitored is recognized in the aforementioned captured image, A processed image is generated by adding an enhanced display to the aforementioned captured image to improve the visibility of the monitored target. The processed image is displayed on the display unit. A program that causes a computer to execute a process to track the monitored target and display it on the display unit, based on an input operation in which the user selects the monitored target to which the aforementioned highlighting has been added, From among the multiple segmented images obtained by dividing the environment into predetermined ranges and capturing them, a processed segmented image is generated by adding the highlighting to one of the segmented images in which the monitored target is captured. Based on the input operation, the processed segmented image is selected and the processed segmented image is displayed on the display unit. While the segmented image is being displayed on the display unit, When the monitored object moves from one of the segmented image captures to another of the segmented image captures, the other segmented image captures is displayed on the display unit. A program that, if a previously tracked target is captured in the image again, automatically tracks the target and displays it on the display unit.

Images