Image processing device and image processing method

Abstract

To provide an image processing device capable of effectively searching for a desired scene from a landscape moving image photographed by a mobile body, and provide an image processing method of the same.SOLUTION: An image processing device of the present invention includes: an acquisition part that acquires a landscape moving image photographed by a mobile body; a moving image generation part that generates an information holding moving image as a landscape moving image obtained by applying importance information based on information in regard to a photographing situation at each point in an imaging period of the landscape moving image to the landscape moving image; and a thumbnail group image generation part that extracts a plurality of thumbnail images on the basis of the importance information from a plurality of frame images constructing the landscape moving image, and generates a thumbnail group image in which the thumbnail images are arranged in a time sequence. The thumbnail group image generation part changes the number of extractions of the thumbnail images per a unit photographing time in accordance with the importance information.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device and a processing method for processing an image. [Background technology]

[0002] 2. Description of the Related Art In recent years, playback devices have become widespread that allow a user to capture a scenic video using a camera mounted on a moving object such as a vehicle and later review the scenes captured from the vehicle.

[0003] For example, Patent Document 1 discloses a display system for a vehicle that displays on a display an instruction display image including a time bar corresponding to the shooting time from the start to the end of shooting of a landscape video captured by an onboard camera mounted on a moving body, a slider that indicates a position on the time bar that can be indicated by the user, and landmark icons that indicate landmarks that existed along the driving route when the landscape video was captured at corresponding positions on the time bar, and further displays on the display a landscape video that is associated with the shooting time that corresponds to the position of the slider on the time bar.

[0004] Furthermore, for example, Patent Document 2 discloses a video search and viewing device that extracts still images from video data, generates thumbnail images, and displays them on a timeline at regular time intervals, making it possible to easily find the part of the video in which a desired scene is recorded. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-32949 [Patent Document 2] Japanese Patent Application Laid-Open No. 2002-281432 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the display system of Patent Document 1, when trying to find the part of a video that shows a desired scene from a video of an external scene shot from a moving object, the user must search for the desired scene while changing the position of the slider or playing the video from near a landmark icon and visually checking the image displayed on the display, so if the video is long, it may take some time to reach the desired scene.

[0007] Furthermore, in the video image search and viewing device of Patent Document 2, for example, if the moving speed of a moving object is slow and the scenery changes little over a given time period, multiple similar scenes or images will be lined up in thumbnail images spaced at regular time intervals, so it may take some time to find a thumbnail image showing the desired scenery.

[0008] The present invention has been made in consideration of the above points, and aims to provide an image processing device and an image processing method that can efficiently search for desired scenes from landscape videos captured from a moving object. [Means for solving the problem]

[0009] The invention described in claim 1 comprises an acquisition unit that acquires a landscape video shot from a moving body, a video generation unit that generates an information-containing video by adding importance information to the landscape video based on information about the shooting conditions at each point in time during the shooting period of the landscape video, and a thumbnail group image generation unit that extracts a plurality of thumbnail images from a plurality of frame images that make up the landscape video based on the importance information and generates a thumbnail group image in which the thumbnail images are arranged in chronological order, wherein the thumbnail group image generation unit changes the number of thumbnail images extracted per unit shooting time according to the importance information.

[0010] Furthermore, the invention described in claim 10 is an image processing method in which an image processing device extracts thumbnail images from an information-containing video to which importance information has been assigned, and is characterized by including an interval determination step of determining an extraction interval for extracting thumbnail images based on the importance information and the number of thumbnail images to be generated, a frame image determination step of determining a plurality of frame images to be extracted as thumbnail images from a plurality of frame images constituting the information-containing video based on the extraction interval, and an extraction step of extracting a plurality of frame images to be extracted as thumbnail images from the information-containing video to generate a thumbnail group image.

[0011] Furthermore, the invention described in claim 11 is an image processing program executed by an image processing device provided in a computer, characterized in that the computer is caused to execute an interval determination process for determining an extraction interval for extracting thumbnail images based on importance information and the number of thumbnail images to be generated, a frame image determination process for determining a plurality of frame images to be extracted as thumbnail images from a plurality of frame images constituting an information-containing video based on the extraction interval, and an extraction process for extracting a plurality of frame images to be extracted as thumbnail images from the information-containing video to generate a thumbnail group image. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a perspective view of an automobile M equipped with an image processing device 10 according to an embodiment of the present invention. [Figure 2] 1 is a view of the rear door of an automobile M equipped with an image processing device 10 according to an embodiment of the present invention, as seen from the rear seat of the automobile M. FIG. [Figure 3] FIG. 2 is a diagram showing an example of an operation acceptance display displayed on the touch panel 17 of the image processing device 10 according to the embodiment of the present application. [Figure 4] 1 is a functional block diagram of an image processing device 10 according to an embodiment of the present invention. [Figure 5] 10 is a flowchart FC1 for generating an information-containing moving image by adding a virtual time code to a scenic moving image captured by an in-vehicle camera CM in an image processing device 10 according to an embodiment of the present application. [Figure 6] 10 shows an example of the elapsed time from the start of shooting a landscape video by the image processing device 10 according to the embodiment of the present application, the frame numbers of a plurality of frames at predetermined intervals, the basic time codes of the plurality of frames, and the virtual time codes. [Figure 7] This figure shows the relationship between the frame numbers of multiple frames that make up the information-containing video in the example of the information-containing video shown in Figure 6, the virtual time codes assigned to each frame number, and the relationship between the thumbnail images that are extracted. [Figure 8] The ten thumbnail images SM1 to SM10 extracted from the scenic video of this embodiment are the time codes, frame numbers, and importance levels associated with the thumbnail images SM1 to SM10. [Figure 9] 10 is a flowchart FC2 when generating thumbnail images in the image processing device 10 according to the embodiment of the present application. [Figure 10A] 9 is a diagram showing an example of a display on the touch panel 17 when the information-containing moving image shown in FIGS. 6, 7, and 8 is played back. FIG. [Figure 10B] 9 is a diagram showing an example of a display on the touch panel 17 when the information-containing moving image shown in FIGS. 6, 7, and 8 is played back. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0013] Examples of the present invention will be described in detail below. [Example]

[0014] FIG. 1 is a perspective view of an automobile M equipped with an image processing device 10 according to an embodiment of the present invention.

[0015] The image processing device 10 is an image processing device that captures a moving image of the scenery around the automobile M using an on-board camera mounted on the automobile M and plays back the captured moving image of the scenery. In the following explanation, a configuration that can display the moving image of the scenery on the window of the automobile M will be described.

[0016] The image processing device 10 is mounted, for example, in the dashboard of an automobile M. The image processing device 10 is configured to be able to play back and output video images that are acquired from an external device or that are stored in the image processing device 10 itself.

[0017] The in-vehicle camera CM serving as the acquisition unit is mounted, for example, on the roof or door panel of the automobile M, and is configured to be able to capture images of the scenery in front of, beside, or behind the automobile M. The in-vehicle camera CM is communicably connected to the image processing device 10, and is configured to be able to transmit captured still images or videos to the image processing device.

[0018] The display 15 as a moving image display unit is configured with a transparent panel such as a transparent liquid crystal or organic EL display, is attached to the window WD of the automobile M, and is capable of displaying images visible from inside the automobile M. The display 15 is connected to the image processing device 10 so as to be able to communicate with the image processing device 10, and is configured to be able to display still images or moving images output from the image processing device 10.

[0019] For example, the display 15 is configured to be able to display, on the front window, door windows, and rear window, moving images of scenery in front of, beside, or behind the automobile M captured by an in-vehicle camera CM. The display 15 may be, for example, a display of a car navigation system installed inside the automobile M.

[0020] FIG. 2 is a view of the rear door of an automobile M equipped with an image processing device 10 according to an embodiment of the present invention, as viewed from the rear seat of the automobile M.

[0021] As described above, the display 15 is configured with a transparent panel such as a transparent liquid crystal or organic EL display, and is attached to the window WD of the automobile M. Therefore, for example, when no video is displayed on the display 15, it is possible to see the outside scenery from inside the automobile M through the display 15.

[0022] The touch panel 17 as an operation receiving unit is a touch panel display attached to the inside of the door panel. For example, the touch panel 17 is a touch panel display including a display 17a such as a liquid crystal panel and a touch pad 17b attached to the display surface of the display and receiving operations by a user's finger or the like.

[0023] The touch panel 17 is communicatively connected to the image processing device 10 and is capable of displaying images output from the image processing device 10. The touch panel 17 is also capable of receiving operations to play, fast forward, or fast rewind a video displayed on the display 15, and transmitting instructions based on the received operations to the image processing device 10.

[0024] Specifically, for example, on the touch panel 17, thumbnail images of the video being played back on the display 15 for each predetermined time period are displayed in order of playback time.

[0025] FIG. 3 is a diagram showing an example of an operation acceptance display displayed on the touch panel 17 of the image processing device 10 according to the embodiment of the present application.

[0026] The operation acceptance display includes, for example, a thumbnail image string SR in which thumbnail images SM1 to SM6 of the scenery video being played back on the display 15 are arranged in chronological order for playback, a button PL for switching between playback and pause, a button FW for a fast-forward operation, a button RW for a fast-rewind operation, a bar BA indicating the current playback time position of the scenery video, and a time display unit TM. Furthermore, the thumbnail images included in the thumbnail image string SR are displayed while scrolling from right to left as the playback time of the scenery video passes.

[0027] 4 is a functional block diagram of an image processing device 10 according to an embodiment of the present application. The image processing device 10 is a device in which an input unit 20, an output unit 30, an operation input unit 40, a storage unit 50, and a video control unit 60 cooperate with each other via a system bus 13, for example.

[0028] The input unit 20 is an interface that acquires data from devices external to the image processing device 10. The input unit 20 is connected to an in-vehicle camera CM provided in the automobile M. For example, the image processing device 10 can capture scenery images from the in-vehicle camera CM via the input unit 20.

[0029] The input unit 20 is also connected to a speed sensor SS that acquires the traveling speed of the automobile, and a GPS receiver GP that can receive information from a Global Positioning System (GPS), which is one of the Global Navigation Satellite Systems (GNSS), that acquires the position information of the automobile. That is, the image processing device 10 is configured to be able to acquire the speed of the automobile M from the speed sensor SS and to acquire GPS information from the GPS receiver GP.

[0030] The output unit 30 is an interface that outputs data to devices external to the image processing device 10. The output unit 30 is connected to the above-mentioned display 15 and the display 17a of the touch panel 17, and is capable of outputting images to these. Specifically, for example, the image processing device 10 outputs a moving image to the display 15 via the output unit, and outputs an operation image that serves as an operation acceptance display for a playback operation to the display 17a.

[0031] The operation input unit 40 is an interface that receives an operation input signal from an external device. The operation input unit 40 is connected to the touchpad 17b and can receive a signal indicating an operation accepted on the operation image on the touchpad 17b.

[0032] The storage unit 50 is configured with, for example, a hard disk drive, an SSD (solid state drive), a flash memory, etc., and stores various programs related to the generation of thumbnail images, the playback of moving images, etc. in the image processing device 10. Note that the various programs may be obtained, for example, from another server device or the like via a network (not shown), or may be recorded on a recording medium and read via various drive devices.

[0033] The various programs stored in the storage unit 50 can be transmitted via a network, and can also be recorded on a computer-readable recording medium and transferred. The storage unit 50 can also store data acquired from outside the image processing device 10, such as landscape video data acquired from the camera CM, and the speed and position information of the automobile M when capturing the landscape video acquired from the speed sensor SS and the GPS receiver GP. The storage unit 50 can also store data generated within the image processing device 10.

[0034] The video control unit 60 is configured with a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and functions as a computer. In the video control unit 60, the CPU reads and executes various programs stored in the ROM and storage unit 50, thereby realizing various functions.

[0035] The video control unit 60 has a video generation unit 63 that adds information to the scenery video acquired via the input unit 20 to generate an information-containing video for playback. The video generation unit 63 reads out the scenery video recorded in the storage unit 50 and data on the speed and position information of the automobile M at each point in time when the scenery video was captured. For each of the multiple frame images that make up the read scenery video, the video generation unit 63 adds information based on the speed or position information of the automobile M at each point in time when each of the multiple frames was captured. For example, the video generation unit 63 determines an importance level for each frame image according to the traveling speed of the automobile M, and adds importance information based on that importance level to the scenery video.

[0036] The importance is determined, for example, based on the traveling speed of the automobile M, the shooting location, the visual appeal of the scenery associated with the position on the map information corresponding to the shooting location, or the shooting direction taken by the in-vehicle camera CM based on the traveling direction of the automobile M.

[0037] For example, when the automobile M is traveling at a low speed or is stopped, the surrounding scenery seen from the automobile hardly changes. Therefore, when determining the importance based on the traveling speed of the automobile, the video generation unit 63 determines that the importance of the scenery when the automobile M is traveling at a low speed or is stopped is low. Conversely, when the automobile is traveling at a high speed, the scenery video changes significantly in a short period of time, so the importance is determined to be high.

[0038] Furthermore, the video generation unit 63 determines that the importance of the vehicle is high when the vehicle is located at a position with a high ranking of attractions on the map database by linking the vehicle's position information obtained by GPS with a pre-recorded map database, for example. Conversely, the video generation unit 63 determines that the importance of the scenery video is low when the vehicle M is located at a position with a low ranking of attractions on the map database.

[0039] Furthermore, the importance of location information may be determined without linking it to a map database. For example, in mountainous areas at high altitudes, the scenery may be very impressive. Therefore, the importance may be determined to be high when positioning within a predetermined range is detected from positioning information such as GPS.

[0040] Furthermore, in a scenery video while the automobile M is traveling, scenery videos taken to the sides in the traveling direction change significantly in a short period of time compared to scenery videos taken in front or behind the traveling direction of the automobile M. Therefore, the video generation unit 63 determines that the importance of scenery videos to the sides in the traveling direction is higher than that of scenery videos taken in front or behind the traveling direction.

[0041] Furthermore, if the GPS receiver GP is unable to acquire GPS information for a predetermined period of time or longer, it is highly likely that the automobile M is traveling through a structure such as a tunnel, and in such a case, the landscape video will continue to have monotonous content. Therefore, the video generator 63 determines that the importance of the landscape video when GPS information is not received is lower than the importance of the landscape video when GPS information is received.

[0042] The above-mentioned criteria for determining the importance can be set arbitrarily. The criteria for determining the importance are not limited to one of the following: the traveling speed of the automobile M, the photographing position, the visual appeal of the scenery associated with the position on the map information corresponding to the photographing position, or the photographing direction taken by the photographing unit relative to the traveling direction of the automobile M. The importance may be determined based on a plurality of criteria.

[0043] In other words, the video generation unit 63 determines the importance based on the moving speed of the automobile M, the shooting location, the visual appeal of the scenery associated with the position on the map information corresponding to the shooting location, the shooting direction based on the moving direction of the automobile M, or the radio wave reception conditions of the satellite system.

[0044] The importance is not limited to the speed or position when the moving image is captured. For example, by comparing each of a plurality of frame images that make up the moving image, a frame with a large change in image between the frame images may be determined to have a high importance.

[0045] Furthermore, for example, the video generation unit 63 generates an information-containing video that is assigned a virtual time code indicating the passage of time different from the time code (hereinafter referred to as the basic time code) that indicates the actual passage of time when the landscape video was shot based on the importance level.

[0046] The virtual time code is calculated from the average value of importance for each predetermined time period, and is importance information in a format that reflects importance over virtual time.

[0047] Specifically, the video generation unit 63 assigns a basic time code and a virtual time code including a different time lapse to each of a plurality of frame images of the scenery video for each predetermined time period, according to the average value of importance calculated for each predetermined time period. That is, the video generation unit 63 functions as an importance information acquisition unit that generates importance and a virtual time code from the scenery video and data on the speed or position information of the automobile M at the time the scenery video was captured. The video generation unit 63 also functions as a video generation unit that assigns the generated virtual time code as importance information to the scenery video to generate an information-containing video.

[0048] The virtual time code is set so that the higher the importance, the closer the time lapse of the basic time code is to the time lapse of the basic time code, and the lower the importance, the earlier the time lapse of the basic time code is to the time lapse of the basic time code.

[0049] Furthermore, the video generating unit 63 can arbitrarily set the predetermined time for calculating the average importance level.

[0050] The information-containing video generated by the video generation unit 63 is stored in the storage unit 50. The importance information assigned to the video may include one or more of the following: the moving speed of the automobile M, the shooting position, the visual appeal of the scenery associated with the position on the map information corresponding to the shooting position, the shooting direction based on the moving direction of the automobile M, the importance based on the radio wave reception status of the satellite system, and a virtual time code.

[0051] The thumbnail generation unit 65, which serves as a thumbnail group image generation unit, extracts frame images at predetermined time intervals based on the virtual time code and the number of thumbnail images to be generated from a plurality of frame images that make up the information-containing moving image recorded in the storage unit 50, and generates thumbnail images to be displayed in the thumbnail display area SR of the touch panel 17. The thumbnail generation unit 65 supplies the generated thumbnail images to the display 17a via the output unit 30, and displays them in the thumbnail image string SR.

[0052] The number of thumbnail images to be generated can be set arbitrarily. For example, a predetermined number may be set for one landscape image, or the number may be set automatically based on the recording time of the captured landscape video.

[0053] The playback control unit 67 reads out information-holding videos recorded in the storage unit 50, and controls playback of the information-holding videos in response to instructions to play, pause, fast-forward, or fast-rewind. The playback control unit 67 supplies the played information-holding videos to the display 15 via the output unit 30. The playback control unit 67 also supplies an operation image corresponding to the information-holding video being played to the display 17a via the output unit 30. The playback control unit 67 acquires an operation signal accepted by the touchpad 17b, and controls playback of the videos based on the operation signal.

[0054] In other words, it comprises an in-vehicle camera CM that acquires a landscape video shot from a moving body, a video generation unit 63 that generates an information-containing video in which a virtual time code based on information about the shooting conditions at each point during the shooting period of the landscape video is assigned to the landscape video, and a thumbnail generation unit 65 that extracts a plurality of thumbnail images from a plurality of frame images that make up the landscape video based on the virtual time code and generates a thumbnail group image in which the thumbnail images are arranged in chronological order, and the thumbnail generation unit 65 changes the number of thumbnail images extracted per unit shooting time according to the virtual time code.

[0055] The image processing device 10 also includes a video generation unit 63 that acquires information regarding the shooting conditions at each point in time during the shooting period of the landscape video, a memory unit 50 that records the information-containing video generated by the video generation unit 63, a playback control unit 67 that reads the information-containing video from the memory unit 50 and controls the playback, fast-forwarding, or fast-rewinding of the landscape video to output the image, and a touch panel 17 that displays a thumbnail image string SR in which thumbnail images generated by the thumbnail generation unit 65 are arranged in order of playback time, and accepts instruction operations including an operation to select a thumbnail image included in the thumbnail image string SR or an operation to scroll the thumbnail image string SR, and the playback control unit 67 performs control based on the instruction operations.

[0056] In addition, the video generation unit 63 generates an information-containing video by assigning a virtual time code, which is a series of virtual time codes based on information about the shooting conditions when each of the multiple frame images was shot, to each of the multiple frame images that make up the shot scenery video, as importance information, and the thumbnail generation unit 65 extracts thumbnail images based on the virtual time codes.

[0057] Note that the thumbnail image may be the extracted frame image itself or a reduced version of the extracted frame image. Below, both the process of extracting frame images from a landscape video and using them as thumbnail images as they are, and the process of reducing the extracted frame images to generate thumbnail images will be described as simply extracting thumbnail images.

[0058] Furthermore, the video generating unit 63 determines the importance of each of the plurality of frame images based on information relating to the shooting conditions, and assigns virtual time codes corresponding to the importance to the plurality of frame images.

[0059] Furthermore, the video generating unit 63 assigns virtual time codes so that the lower the importance, the faster the time indicated by the virtual time codes in the group of frame images passes.

[0060] 5 shows a flowchart FC1 for generating an information-containing video by adding a virtual time code to a landscape video captured by an in-vehicle camera CM in the image processing device 10 according to the embodiment of the present application. Here, an example will be described in which the importance of the landscape image is set according to the traveling speed of the automobile M.

[0061] First, in step S101, the video generation unit 63 determines whether or not the shooting of the scenery video has been completed (step S101). This determination may be made, for example, by accessing the camera CM and checking whether or not the video shooting has been completed. Alternatively, this determination may be made, for example, by checking whether or not the reception of video data from the camera CM has been completed.

[0062] If it is determined that the shooting of the landscape video has ended (step S101: Yes), the video generation unit 63 reads out the landscape video recorded in the memory unit 50 and the traveling speed of the automobile M acquired by the speed sensor SS during the shooting period (step S102).

[0063] Next, the video generating unit 63 assigns to each of the plurality of frame images that make up the read scenery video the traveling speed of the automobile M at the time the frame was captured (step S103).

[0064] The video generation unit 63 determines the importance according to the traveling speed of the automobile M for each of the multiple frames constituting the scenery video to which the traveling speed of the automobile M has been assigned, and further assigns the importance to the scenery video (step S104).

[0065] Next, the video generating unit 63 divides the scenery video in which importance has been assigned to each frame into predetermined time intervals, and calculates the average importance for each predetermined time interval (step S105).

[0066] Thereafter, the video generating unit 63 assigns a virtual time code including a time lapse different from the basic time code in accordance with the average value of the importance for each predetermined time period (step S106).

[0067] After step S106 is completed, the moving image generating unit 63 records the scenery moving image to which the virtual time code has been assigned as an information-containing moving image in the storage unit 50 (step S107).

[0068] By performing the above process, an information-containing video with importance and a virtual time code is generated from the landscape video captured by the in-vehicle camera CM and the traveling speed of the automobile M obtained by the speed sensor SS.

[0069] In this embodiment, it has been described that the assignment of the traveling speed of the automobile M, the assignment of the importance, and the assignment of the virtual time code to the scenery video are performed after the scenery video is shot.

[0070] However, the assignment of the traveling speed of the automobile M, the assignment of importance, and the assignment of a virtual time code to the scenery video may be performed while the scenery video is being filmed. When performed while filming, each time a frame image of the scenery video is supplied, the assignment of the traveling speed of the automobile M at the time of that frame image, and the determination and assignment of importance are performed. Also, each time the recording time of the scenery video exceeds a predetermined time for calculating the average importance, the average importance is calculated and a virtual time code is assigned.

[0071] Next, the virtual time code assigned to the information-containing moving image will be described with reference to FIG.

[0072] 6 shows an example of the elapsed time from the start of landscape video shooting by the image processing device 10 according to an embodiment of the present application, the frame numbers of multiple frames for each predetermined time period, and the basic time codes of the multiple frames. It also shows the average importance for each predetermined time period calculated by the video generating unit 63, the multiplication factor of the virtual time code with respect to the basic time code, and the assigned virtual time code.

[0073] In this embodiment, we will explain the case where a 600-second landscape video captured by an in-vehicle camera CM has a shooting frame rate of 30 fps, a predetermined time period for calculating the average importance level is 60 seconds, and importance levels are set to four levels from 0 to 3 according to the driving speed.

[0074] In addition, the multiplier of the virtual time code will be described as a setting in which the virtual time code is assigned at a time code interval equal to the basic time code for importance level 3, 1 / 2 the basic time code for importance level 2, 1 / 3 the basic time code for importance level 1, and 1 / 5 the basic time code for importance level 0.

[0075] The basic time code and the virtual time code progression for importance level 3 progress in accordance with the frame rate of the captured landscape video, so they progress at a rate of 1 / 30 second (approximately 0.033 seconds) per frame. The virtual time code progression for importance level 2 progresses at a rate of 1 / 60 second (approximately 0.017 seconds) per frame, the virtual time code progression for importance level 1 progresses at a rate of 1 / 90 second (approximately 0.011 seconds) per frame, and the virtual time code progression for importance level 0 progresses at a rate of 1 / 150 second (approximately 0.007 seconds) per frame.

[0076] In other words, importance level 3 assigns a virtual time code whose time lapse rate is the same as that of the basic time code, importance level 2 assigns a virtual time code whose time lapse rate is approximately twice that of the basic time code, importance level 1 assigns a virtual time code whose time lapse rate is approximately three times that of the basic time code, and importance level 0 assigns a virtual time code whose time lapse rate is approximately five times that of the basic time code.

[0077] In Figure 6, the average value of the calculated importance for all frames between 0 and 180 seconds after the start of shooting is 3. Therefore, the virtual time code is assigned at the same rate as the basic time code, that is, 1 / 30 second per frame.

[0078] Furthermore, the average value of the calculated importance for the time period from 180 to 300 seconds elapsed since the start of shooting is 1. Therefore, the virtual time code is assigned at a speed of 1 / 90 seconds per frame, which is one-third the speed of the basic time code. In other words, for the 120 seconds from 180 to 300 seconds elapsed since the start of shooting, the basic time code progresses at the same speed as the actual shooting time for this period, but the virtual time code is assigned at a speed of 40 seconds per frame for the actual shooting time of 120 seconds for this period.

[0079] Furthermore, the average calculated importance level for the time period from 300 to 420 seconds elapsed since the start of shooting is 2. Therefore, the virtual time code is assigned at a rate of 1 / 60 seconds per frame, which is half the rate of the basic time code. In other words, for the 120 seconds from 300 to 420 seconds elapsed since the start of shooting, the virtual time code is assigned at a rate of 60 seconds per frame for the actual shooting time of 120 seconds during this period.

[0080] Furthermore, the average calculated importance level between 420 and 480 seconds after the start of shooting is 0. Therefore, the virtual time code is assigned at a rate of 1 / 150 seconds per frame, which is 1 / 5 the speed of the basic time code. In other words, during the 60 seconds from 420 to 480 seconds after the start of shooting, the virtual time code is assigned at a rate of 12 seconds per frame relative to the actual shooting time of 60 seconds.

[0081] Furthermore, the average value of the calculated importance between 480 and 600 seconds after the start of shooting is 2. The progression speed of the virtual time code during this time is the same as in the case of importance 2 above, so a description thereof will be omitted.

[0082] That is, in a 600-second landscape video shot at 30 fps by a car-mounted camera commercial, the basic time code is constant at 1 / 30 second per frame, so the basic time code is assigned a time code of 0 to 599.967 seconds for frame numbers 1 to 18000. In contrast, the virtual time code progresses at a rate that changes depending on the importance of each predetermined time period, so the virtual time code is assigned a time code of 0 to 351.967 seconds for frame numbers 1 to 18000.

[0083] Figure 7 shows the relationship between the frame numbers of the multiple frames that make up the information-containing video in the example of the information-containing video shown in Figure 6, the virtual time codes assigned to each frame number, and the relationship between the thumbnail images that are extracted.

[0084] The horizontal axis indicates the frame numbers of the multiple frames that make up the information-containing video. It also indicates the elapsed time (basic time code) from the start of shooting that corresponds to each frame number. The vertical axis indicates the virtual time code, and indicates the virtual time code that corresponds to the frame number on the horizontal axis.

[0085] In this embodiment, a case where 10 thumbnail images are extracted from the information-containing moving image shown in FIG. 6 will be described.

[0086] As shown in FIG. 7, the section from frame numbers 1 to 5400 (0 to 180 seconds elapsed since the start of shooting) has a calculated importance of 3, so the virtual time code per frame is assigned with the same elapsed time as the basic time code.

[0087] For the section from frame numbers 5401 to 9000 (180 to 300 seconds elapsed since the start of shooting), the calculated importance is 1, so the virtual time code per frame is assigned with an elapsed time that is 1 / 3 of the basic time code.

[0088] For the section from frame numbers 9001 to 12600 (300 to 420 seconds elapsed from the start of shooting), the calculated importance is 2, so the virtual time code per frame is assigned with an elapsed time that is half the basic time code.

[0089] For the section from frame numbers 12601 to 14400 (420 to 480 seconds elapsed since the start of shooting), the calculated importance is 0, so the virtual time code for each frame is assigned with an elapsed time that is 1 / 5 of the basic time code.

[0090] For the section from frame numbers 14401 to 18000 (480 to 600 seconds elapsed since the start of shooting), the calculated importance is 2, so the virtual time code for each frame is assigned with an elapsed time that is half the basic time code.

[0091] As a result, for the information-containing moving image in this embodiment, the relationship between the virtual time code and the multiple frames that make up the information-containing moving image is shown as a solid line in the figure.

[0092] The virtual time code is assigned so that the lower the importance, the faster the virtual time code progresses. Therefore, as shown in Figure 7, the lower the importance, the shorter the time elapsed per frame, and the smaller the slope of the virtual time code.

[0093] The thumbnail generating unit 35 extracts frame images for each predetermined time of the virtual time code as thumbnail images based on the virtual time code.

[0094] Therefore, when generating 10 thumbnail images from a landscape video in this embodiment, since the virtual time code of the information-holding video is assigned a time code from 0 to 351.967 seconds, the thumbnail generation unit 65 extracts 10 frame images at intervals of approximately 35.2 seconds starting from 0 second on the virtual time code. In other words, the thumbnail generation unit 65 extracts frame images at regular intervals of approximately 35.2 seconds on the vertical axis in the figure.

[0095] The frame images extracted as thumbnail images are selected at regular intervals of approximately 35.2 seconds from the virtual time code starting at 0 seconds, 35.2 seconds, 70.4 seconds, 281.6 seconds, and 316.8 seconds. Because the virtual time code progresses at different speeds depending on the importance of each specified time period, the basic time codes and frame numbers of the extracted thumbnail images are not set at regular intervals.

[0096] Therefore, as shown in FIG. 7, the lower the importance, the fewer the number of thumbnail images extracted per unit shooting time.

[0097] That is, according to the present invention, the progression speed of the virtual time code is changed according to the importance of each predetermined time period, so that it is possible to change the number of thumbnail images extracted per unit shooting time according to the importance.

[0098] FIG. 8 shows the time codes, frame numbers and importance levels associated with the ten thumbnail images SM1 to SM10 extracted from the scenic video of this embodiment.

[0099] The upper part of Fig. 8 shows information about thumbnail images generated from frame images extracted based on a conventional basic time code as a comparative example, while the lower part of Fig. 8 shows information about thumbnail images generated from frame images extracted based on a virtual time code according to the present invention. The lower part of Fig. 8 also shows basic time codes corresponding to the frame numbers of thumbnail images extracted based on the virtual time code.

[0100] 8, when extracting thumbnail images based on the basic time code, the thumbnail generation unit 65 extracts frame images at fixed intervals of 60 seconds when the basic time code is 0, 60, 120, . . ., 480, and 540 seconds. The frame numbers of the extracted thumbnail images are 1, 1801, 3601, . . ., 14401, and 16201, which are fixed intervals of 1 to 1800. The importance of the extracted frame images is 3 for the first to third frame images, 1 for the fourth and fifth frame images, 2 for the sixth and seventh frame images, 0 for the eighth frame image, and 2 for the ninth and tenth frame images.

[0101] 8, when extracting thumbnail images based on the virtual time code, the thumbnail generation unit 65 extracts frame images at regular intervals of 35.2 seconds when the virtual time code is 0, 35.2, 70.4, . . ., 281.6, and 316.8 seconds as thumbnail images. The frame numbers of the extracted thumbnail images are 1, 1056, 2112, 3168, 4224, 5280, 8211, 10586, 12845, and 15890. The importance levels of the extracted frame images are 3 for the first to sixth frame images, 1 for the seventh frame image, 2 for the eighth frame image, 0 for the ninth frame image, and 2 for the tenth frame image.

[0102] That is, the thumbnail generation unit 65 extracts fewer thumbnail images from time segments of low importance that are determined to have few highlights in the captured landscape video, and extracts more thumbnail images from time segments of high importance that are determined to have many highlights.

[0103] Furthermore, as indicated by the basic time codes corresponding to the frame images of thumbnail images extracted based on the virtual time code, the intervals between the shooting times of the extracted thumbnail images are longer in sections of low importance and shorter in sections of high importance. That is, the time intervals between extraction of thumbnail images or frame images used to generate those thumbnail images are longer in sections of low importance, and the time intervals between extraction of thumbnail images or frame images used to generate those thumbnail images are shorter in sections of high importance.

[0104] This allows many thumbnail images near desired scenes with high importance in the captured scenery video to be extracted, making it possible to efficiently search for desired scenes from the captured scenery video.

[0105] Note that instead of setting the importance of a scenic image according to the traveling speed of the automobile M, the video generation unit 63 may set the importance of a scenic image based on the shooting position, the impressiveness of the scenery associated with the position on the map information corresponding to the shooting position, the shooting direction based on the moving direction of the automobile M, or the radio wave reception status of the satellite system. Alternatively, the video generation unit 63 may set the importance of a scenic image based on a combination of multiple elements from among the traveling speed, the shooting position, the impressiveness of the scenery associated with the position on the map information corresponding to the shooting position, the shooting direction based on the moving direction of the automobile M, and the radio wave reception status of the satellite system.

[0106] FIG. 9 shows a flowchart FC2 when generating thumbnail images in the image processing device 10 according to the embodiment of the present application.

[0107] First, the playback control unit 67 determines whether or not a playback instruction for a scenery video has been issued. This determination may be made, for example, based on whether or not a playback instruction operation for a scenery video has been accepted on the touch panel 17 (step S201).

[0108] Next, the playback control unit 67 reads out the information-containing moving image that has been instructed to be played and that is recorded in the storage unit 50 (step S202).

[0109] As an interval determination step, the thumbnail generation unit 65 determines the extraction interval of the virtual time code for extracting thumbnail images from the virtual time code of the information-containing video read by the playback control unit 67 and the number of thumbnail images to be generated (step S203).

[0110] Next, as a frame image determination step, the thumbnail generating unit 65 determines the frame number of the thumbnail image from the thumbnail image extraction interval of the determined virtual time code (step S204).

[0111] Next, in the extraction step, the thumbnail generating unit 65 extracts the frame image of the determined frame number as a thumbnail image (step S205).

[0112] Thereafter, the thumbnail generating unit 65 arranges the extracted thumbnail images in chronological order and supplies them to the output unit 30 (step S206).

[0113] After the process of step S206 is completed, the thumbnail generating unit 65 ends the process, and thereafter the playback control unit 67 executes the playback process of the information-holding moving image.

[0114] By the above process, it is possible to generate a group of thumbnail images that includes many thumbnail images of sections with high importance and few thumbnail images of sections with low importance.

[0115] In other words, the image processing method of the present invention is an image processing method in which an image processing device 10 extracts thumbnail images from an information-containing video to which a virtual time code has been assigned, and includes an interval determination step of determining an extraction interval for extracting thumbnail images based on the virtual time code and the number of thumbnail images to be generated, a frame image determination step of determining a plurality of frame images to be extracted as thumbnail images from a plurality of frame images constituting the information-containing video based on the extraction interval, and an extraction step of extracting a plurality of frame images to be extracted as thumbnail images from the information-containing video to generate a thumbnail group image.

[0116] 10A and 10B are diagrams showing examples of displays on touch panel 17 when the information-containing moving images shown in FIGS. 6, 7, and 8 are played back.

[0117] Fig. 10A shows the operation acceptance display on touch panel 17 during playback of a section with an importance level of 3, and Fig. 10B shows the operation acceptance display on touch panel 17 during playback of a section with an importance level of 1. The length of the arrows in the figures indicates the scrolling speed. The times displayed in the lower left corner of thumbnail images SM1 to SM10 are the playback times (basic time codes) corresponding to the frames of the displayed thumbnail images SM1 to SM10.

[0118] During playback, the playback control unit 67 controls the playback speed, the playback time display in the time display unit TM, and the scrolling speed of the thumbnail image string SR based on the basic time code.

[0119] For example, the scrolling speed during playback in the section of thumbnail image SM3 in Figure 10A, which has an importance level of 3, is approximately 35 seconds, from the beginning of thumbnail image SM3, which has a basic time code of 70.4 seconds, to the beginning of thumbnail image SM4, which has a basic time code of 105.6 seconds.

[0120] Furthermore, the scrolling speed during playback in the section of thumbnail image SM6, which has an importance level of 1 in Figure 10B, is approximately 79 seconds from the beginning of thumbnail image SM7, which has a basic time code of 273.7 seconds, to the beginning of thumbnail image SM8, which has a basic time code of 352.9 seconds.

[0121] In other words, when the original frame image of thumbnail image SM4 is displayed as a still image that makes up the video being displayed on display 15, the scrolling speed of the thumbnail image string SR is controlled so that thumbnail image SM4 comes close to the bar BA that indicates the current playback time position, and when the original frame image of thumbnail image SM8 is displayed as a still image that makes up the video being displayed on display 15, thumbnail image SM8 comes close to the bar BA that indicates the current playback time position.

[0122] That is, the playback time (basic time code interval) of a video corresponding to a thumbnail image with low importance is longer than the playback time of a thumbnail image with high importance. Therefore, the scrolling speed of the thumbnail image string SR is slower when playing a video with low importance, and faster when playing a video with high importance.

[0123] Furthermore, in sections with low importance, thumbnail images displayed in the thumbnail image string SR are displayed in small numbers at long time intervals relative to the recording time of the captured landscape video, and in sections with high importance, thumbnail images are displayed in large numbers at short time intervals relative to the recording time of the captured landscape video.

[0124] When a user searches for a desired scene from a captured landscape video, the suitable shooting time intervals between multiple thumbnail images to be displayed as the thumbnail image sequence SR will differ depending on whether the user has an idea of ​​where in the landscape video the desired scene is recorded or not. In the former case, a thumbnail image sequence SR with thumbnail images around the already-estimated shooting time and with a relatively short shooting time interval between the thumbnail images will be effective, while in the latter case, it is first necessary to have an idea of ​​where in the landscape video the desired scene is recorded, so a thumbnail image sequence SR with a relatively long shooting time interval between the thumbnail images will be effective.

[0125] To address this, it is preferable that the image processing device 10 be configured to allow the user to change the shooting time interval between adjacent thumbnail images among multiple thumbnail images displayed in the thumbnail image string SR as part of the operation reception display.

[0126] Specifically, the image processing device 10 displays an operation icon for allowing the user to specify the time scale of the thumbnail image sequence SR, included in the operation acceptance display on the touch panel 17, and the thumbnail generation unit 65 determines the extraction interval of the virtual time code for extracting thumbnail images to be displayed as the thumbnail image sequence SR, according to the time scale specified by the user's operation. This makes it possible to arbitrarily change the time scale of the thumbnail image sequence, thereby generating a thumbnail image sequence that is effective when searching for a desired scene within a limited time range, or when searching for a desired scene within a relatively long time range.

[0127] In the image processing device 10 of the above-described embodiment, a method has been described in which frame images are extracted from an information-containing video after a playback instruction is issued, and thumbnail images are generated. However, the generation of thumbnail images is not limited to this. For example, after shooting of a landscape video is completed and the video generation unit 63 generates an information-containing video by assigning a virtual time code to the landscape video (steps S101 to S106 in FIG. 5), the thumbnail generation unit 65 may then perform the processing of steps S202 to S205 in FIG. 9 on the information-containing video, thereby recording the information-containing video and thumbnail images in the storage unit 50. In this way, by generating and recording thumbnail images in advance, it is possible to quickly execute the playback process of the landscape video during playback.

[0128] In the embodiment, the image processing device 10 is described in which the importance of a scenic video captured by an on-board camera CM mounted on an automobile M is determined according to the traveling speed and shooting position of the automobile M, and the thumbnail generation unit 65 variably changes the number of thumbnail images to be generated per unit shooting time according to the determined importance. However, the image processing device according to the present invention is not limited to scenic videos captured from an automobile.

[0129] For example, a video may be generated by adding importance to a video shot with a smartphone, a handy camera, etc. Furthermore, the shot information-containing video may be recorded on an external recording device such as a USB memory or a HDD, and then processed and played back on a device such as a PC.

[0130] In this embodiment, the average importance level is calculated for each section of the basic time code, and a virtual time code is assigned to each section. However, the method of assigning virtual time codes is not limited to this. For example, the time lapse for each frame of a landscape video to which importance levels have been assigned may be assigned according to the importance level. This makes it possible to efficiently generate thumbnail images of sections with high importance, even if the importance level changes significantly within a section of the basic time code.

[0131] In this embodiment, the image processing device 10 generates one piece of video data by assigning a virtual time code to a landscape video and performs video processing on the video data. However, the playback control method is not limited to generating one piece of video data. For example, a landscape video without an importance level or a virtual time code and information data indicating frame numbers and virtual time codes corresponding to the frame numbers may be generated from the landscape video. The thumbnail generation unit 65 may determine frame numbers to be extracted as thumbnail images from the information data, and extract frame images of the frame numbers from the landscape video as thumbnail images. This reduces the file data size of the landscape video.

[0132] Furthermore, in this embodiment, a method has been described in which importance is determined from the traveling speed of automobile M and a virtual time code is generated from that importance, but the method of generating a virtual time code is not limited to this. For example, multiple thresholds may be set for the average traveling speed of automobile M for each predetermined time period, and virtual time codes may be set according to the multiple thresholds for the average traveling speed. This makes it possible to generate a virtual time code directly from the traveling speed of automobile M for each predetermined time period.

[0133] In other words, it is possible to assign importance information indicating specific highlights to each of the multiple frame images that make up the captured image, and to variably change the number of thumbnail images generated per unit of shooting time depending on the importance. [Explanation of symbols]

[0134] 10 Image processing device 15 Display 17 Touch Panel 20 Input section 30 Output section 40 Operation input section 50 Storage section 60 Video control section 63 Video Generation Unit 65 Thumbnail generation section 67 Playback control unit

Claims

1. an acquisition unit that acquires video information to which importance information is assigned for each frame image that constitutes the video; an extracting unit that extracts a part of the frame images from the video information as thumbnail images; a thumbnail image generating unit that generates a group of thumbnail images by arranging the plurality of thumbnail images extracted by the extracting unit in time series; Equipped with The image processing device is characterized in that the extraction unit determines the number of thumbnail images to be generated, which is set in advance, based on user operation, or based on the recording time of the video, and the extraction interval for extracting the thumbnail images based on the importance information.

2. 2. The image processing device according to claim 1, The image processing device is characterized in that the extraction unit determines the extraction interval so that the extraction time interval of the thumbnail images corresponding to sections of the video with high importance indicated by the importance information is shorter than the extraction time interval of the thumbnail images corresponding to sections with low importance.

3. 3. The image processing device according to claim 1, The image processing device is characterized in that the extraction unit determines the extraction interval so that the higher the importance indicated by the importance information, the greater the number of thumbnail images extracted per unit shooting time of the video.

4. 4. The image processing device according to claim 1, The image processing device is characterized in that the extraction unit determines the extraction interval based on a virtual time code that indicates the passage of time, which is assigned to each frame image as the importance information and is different from the time code that indicates the actual passage of time when the video was shot.

5. 5. The image processing device according to claim 4, An image processing device characterized in that the virtual time code is assigned so that the lower the importance level indicated by the importance information in the video, the faster the time passes compared to the actual time passing when the video was shot.

6. 6. The image processing device according to claim 1, 10. An image processing device according to claim 9, wherein the importance indicated by the importance information is determined based on a moving speed of a shooting device for the video.

7. 6. The image processing device according to claim 1, 10. An image processing device according to claim 9, wherein the importance indicated by the importance information is determined based on location information of a shooting device for the video.

8. 8. The image processing device according to claim 7, 10. An image processing device according to claim 9, wherein the importance indicated by the importance information is determined based on information corresponding to a position on a map compared with the position of the image capturing device.

9. 6. The image processing device according to claim 1, The image processing device according to claim 1, wherein the importance indicated by the importance information is determined based on a shooting direction relative to a moving direction of a shooting device for the video.

10. An image processing method in which an image processing device generates a group of thumbnail images from moving image information in which importance information is assigned to each frame image constituting a moving image, the method comprising: an extraction step of extracting a part of the frame images from the video information as thumbnail images; a thumbnail image generating step of generating a group of thumbnail images by arranging the plurality of thumbnail images extracted in the extracting step in time series; Including, An image processing method characterized in that, in the extraction step, the number of thumbnail images to be generated is determined based on a predetermined setting, a setting based on a user operation, or a setting based on the recording time of the video, and an extraction interval for extracting the thumbnail images based on the importance information.

11. An image processing program executed by an image processing device including a computer, The computer an acquisition means for acquiring video information to which importance information is assigned for each frame image constituting the video; an extracting means for extracting a part of the frame images from the video information as thumbnail images; and a thumbnail image generating unit that generates a group of thumbnail images in which the plurality of thumbnail images extracted by the extracting unit are arranged in time series; The image processing program is characterized in that the extraction means determines the number of thumbnail images to be generated, which is set in advance, based on user operation, or based on the recording time of the video, and the extraction interval for extracting the thumbnail images based on the importance information.

12. 12. A computer-readable recording medium storing the image processing program according to claim 11.

Images