Image processing device and image processing method

Abstract

To provide an image processing device and image processing method which allow a user to preferably look back on an important scene during travel.SOLUTION: An image processing device 10 comprises an image acquisition unit 11 which acquires image data of the periphery of a vehicle. The image processing device 10 comprises a voice acquisition unit 12 which acquires in-cabin voice data in a cabin of the vehicle. The image processing device 10 comprises a navigation information acquisition unit 16 which acquires navigation information of the vehicle. The image processing device 10 comprises a recording control unit 17 which records the image data in a period including a period in which an event occurs in an image recording unit 21 in a case of the occurrence of the event in which it is determined that a relation degree between at least one of the image data and in-cabin voice data and the navigation information is equal to or greater than a predetermined threshold.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] It is known that image (including still and moving image) data is recorded using navigation information of a vehicle. For example, Patent Document 1 discloses an in-vehicle device that records images related to a digital broadcast received in a vehicle when the broadcast contains a predetermined keyword such as a destination name or a stopover name. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-017514 A Summary of the Invention [Problem to be solved by the invention]

[0004] There is a demand for saving important scenes during travel as a record of the journey and looking back on them later. However, the in-car device described in the above-mentioned Patent Document 1 does not record important scenes during travel.

[0005] The present invention has been made to solve the above problems, and provides an image processing device and an image processing method that enable a user to conveniently look back on important scenes while moving. [Means for solving the problem]

[0006] The image processing device of the present invention includes an image acquisition unit that acquires image data of a range including the surroundings of the vehicle, an audio acquisition unit that acquires indoor audio data within the vehicle's cabin, a navigation information acquisition unit that acquires navigation information for the vehicle, and a recording control unit that, when an event occurs in which it is determined that the degree of association between at least one of the image data and the indoor audio data and the navigation information is equal to or greater than a predetermined threshold, records the image data for a period including the period during which the event occurs in an image recording unit.

[0007] The image processing method of the present invention includes a step of acquiring image data of the surroundings of a vehicle, a step of acquiring indoor audio data of the interior of the vehicle, a step of acquiring navigation information of the vehicle, and, when an event occurs in which it is determined that the degree of association between at least one of the image data and the indoor audio data and the navigation information is equal to or greater than a predetermined threshold, a step of recording the image data for a period including the period during which the event occurs in an image recording unit. Effect of the Invention

[0008] According to the present invention, it is possible to provide an image processing device and an image processing method that enable a user to conveniently look back on important scenes while moving. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of a configuration of an image processing system according to a first embodiment. [Diagram 2] FIG. 4 is a diagram showing an example of a data structure of a related string table according to the first embodiment. [Diagram 3] 5 is a flowchart showing an example of a processing procedure of an image processing method according to the first embodiment. [Figure 4] 11 is a flowchart showing an example of a processing procedure of an image processing method according to a second embodiment. [Diagram 5] FIG. 11 is a block diagram showing an example of the configuration of an image processing system according to a third embodiment. [Figure 6]13 is a flowchart showing an example of a processing procedure of an image processing method according to a third embodiment. [Figure 7] 13 is a flowchart showing an example of a processing procedure of an image processing method according to a fourth embodiment. [Figure 8] FIG. 13 is a block diagram showing an example of the configuration of an image processing system according to a fifth embodiment. [Figure 9] FIG. 13 is a diagram showing an example of a data structure of an associated aspect table according to the fifth embodiment. [Figure 10] FIG. 13 is a diagram showing an example of a data structure of an image processing mode table according to the fifth embodiment. [Figure 11] 13 is a flowchart showing an example of a processing procedure of an image processing method according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. In addition, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems. For clarity of explanation, the following description and drawings are omitted and simplified as appropriate. In each drawing, the same elements are given the same reference numerals.

[0011] <Embodiment 1> First, a first embodiment of the present invention will be described with reference to Figs. 1 to 3. Fig. 1 is a block diagram showing an example of a configuration of an image processing system 1 according to the first embodiment. The image processing system 1 shown in Fig. 1 is a computer system that generates a digest image from images captured around a vehicle.

[0012] The image processing system 1 includes an image processing device 10, a camera 30, and a microphone 40. Here, the image processing device 10, the camera 30, and the microphone 40 configure an in-vehicle equipment system including in-vehicle equipment such as a drive recorder, a driver monitor, or an in-cabin monitor.

[0013] The camera 30 is mounted at an arbitrary position on the vehicle and captures an area including the surroundings of the vehicle. The camera 30 captures the scenery inside the vehicle (inside the vehicle) as well as the scenery outside the vehicle, and generates image data of the captured images. The camera 30 is communicably connected to the image processing device 10. The camera 30 generates image data at, for example, 30 frames per second (30 fps), and supplies the generated image data to the image processing device 10 every 1 / 30th of a second.

[0014] The microphone 40 is mounted at an arbitrary position in the vehicle, and captures voices within the vehicle cabin to generate voice data. As an example, the microphone 40 captures the voices of passengers sitting in the vehicle seats. The microphone 40 is also communicably connected to the image processing device 10. The microphone 40 supplies the generated voice data to the image processing device 10 in the same manner as the camera 30. The microphone 40 is not limited to capturing voices within the vehicle cabin, and may capture voices outside the vehicle and supply them to the image processing device 10.

[0015] The image processing device 10 is a computer or hardware installed in association with an in-vehicle device. The image processing device 10 may be built into the in-vehicle device or may be disposed externally. The image processing device 10 may also be installed at a location away from the vehicle and communicate with the camera 30 and the microphone 40 via wireless communication means. The image processing device 10 generates a digest image from image and audio data acquired from the camera 30 and the microphone 40, respectively. The image processing device 10 may be configured to include the camera 30 and the microphone 40.

[0016] The image processing device 10 includes an image acquisition unit 11, a voice acquisition unit 12, a voice recognition unit 13, a navigation information acquisition unit 16, a recording control unit 17, and a storage unit 20.

[0017] The image acquisition unit 11 acquires image data, which is data of a landscape image to be photographed, from the camera 30. The image acquisition unit 11 may be an interface such as an input terminal or an input circuit connected to the camera 30 by wire, or may be a communication receiving unit connected to the camera 30 by wireless. The image acquisition unit 11 supplies the acquired image data to the recording control unit 17.

[0018] The voice acquisition unit 12 acquires indoor voice data, which is data of voice within the vehicle cabin, from the microphone 40. The voice acquisition unit 12 may be an interface such as an input terminal or an input circuit connected to the microphone 40 by wire, or may be a communication receiving unit connected to the microphone 40 by wireless. The voice acquisition unit 12 supplies the acquired indoor voice data to the voice recognition unit 13.

[0019] The voice recognition unit 13 performs voice recognition processing on the indoor voice data supplied from the voice acquisition unit 12, and converts the indoor voice data into text data including character strings. The voice recognition unit 13 may use a known voice data text conversion technology.

[0020] The navigation information acquisition unit 16 acquires navigation information of the vehicle (hereinafter referred to as navigation information). The navigation information includes, for example, at least one of destination information, waypoint information, user registration information, and driving history information. The navigation information may also include map information, current position information, guided route information, traffic information such as congestion information and accident information between the current position and the destination, distance information, or arrival time information. For example, the navigation information acquisition unit 16 acquires navigation information from a navigation device (not shown) installed in the vehicle, or a navigation device (not shown) such as a smartphone that implements a navigation application. The navigation information acquisition unit 16 may be configured to include a navigation device, or may be configured separately. The navigation information acquisition unit 16 supplies the acquired navigation information to the recording control unit 17. The navigation information is preferably information related to the route traveled by the user, such as destination information, waypoint information, current position information, guided route information, and traffic information between the current position and the destination, among the above.

[0021] When the vehicle interior audio data and the navigation information are related to each other, the recording control unit 17 determines that an event has occurred, and records image data of the event occurrence period in the image recording unit 21. Here, the event occurrence period refers to a period including the period during which the event actually occurs and a predetermined period (e.g., 30 seconds) before and after the event occurrence. However, the event occurrence period is not limited to this, and may include either the period before or after the event occurrence instead of the period before and after the event occurrence, or may not include these periods.

[0022] The recording control unit 17 includes a determination unit 18 . The determination unit 18 calculates the degree of association depending on whether or not the text data converted by the voice recognition unit 13 includes a character string related to the navigation information. A character string related to the navigation information is called a related character string, and may be a character string specified by the related character string table 22 in the storage unit 20. The determination unit 18 then determines whether or not the degree of association between the text data and the navigation information is equal to or greater than a predetermined threshold, and if the degree of association is equal to or greater than the predetermined threshold, determines that an event has occurred.

[0023] The storage unit 20 is a storage medium or storage device that stores various information necessary for processing by the image processing device 10. For example, the storage unit 20 stores an associated character string table 22. The storage unit 20 may include an image recording unit .

[0024] The image recording unit 21 is a storage area for storing image data during an event occurrence period supplied from the recording control unit 17. The image recording unit 21 may store image data supplied from the recording control unit 17 in association with corresponding indoor audio data. The image recording unit 21 is, for example, a storage medium such as an SD card or a hard disk drive. At this time, the recording control unit 17 preferably records the image data in the image recording unit 21 in a recording format that prohibits overwriting. For example, when the image processing device 10 is mounted on a drive recorder, the recording control unit 17 may record in the image recording unit 21 in a recording mode different from the recording modes of a normal drive recorder, such as continuous recording (overwriting), abnormality recording (recording with overwriting prohibited when an impact is detected, etc.), and manual recording. The image recording unit 21 is preferably a recording area different from the recording area of ​​the normal recording mode. It is preferable that the recording control unit 17 records in a manner that allows images recorded in the abnormality recording mode and images stored in the image recording unit 21 to be distinguished from each other. This makes it easy to view the recording file after moving. In order to facilitate browsing, it is preferable to identify the recording files by giving them folder names and file names according to the recording mode, but this is not limited to this. Note that the drive recorder does not need to be equipped with all or some of the above-mentioned recording modes.

[0025] In the first embodiment, the image recording unit 21 is included in the image processing device 10, but instead, it may be included in an external device (not shown) connected to be able to communicate with the image processing device 10. In this case, the recording control unit 17 transmits image data and indoor audio data to the external device, thereby causing the data to be stored in the image recording unit 21 of the external device.

[0026] FIG. 2 is a diagram showing an example of the data structure of the related string table 22 according to the first embodiment. The related string table 22 manages various related strings corresponding to navigation information. In the first embodiment, the navigation information may be information indicating a departure point, a destination, a waypoint, a current location, a registered location of the navigation device, or a destination or waypoint in the travel history, and these locations are collectively referred to simply as "locations". The related strings may indicate at least one of the name of the location (location name), the place name where the location is located (location name), a famous product, famous place, or famous facility name or event related to the location, the nearest station, and a keyword according to the use of the facility related to the location. In this figure, the related strings include, in addition to these, a navigation-registered location name indicating a location or facility registered by a user. For example, the navigation-registered location name may be an abbreviation or nickname of the location name or facility name, and may be, for example, a private name such as "grandma's (house)" or "head office". Note that the related strings are not limited to these, and may also include traffic congestion information and the distance between the current location and the location.

[0027] First, the determination unit 18 refers to the related character string table 22 based on the acquired navigation information, and extracts place names related to the navigation information such as the set destination and waypoints, the current location, and points on the guided route, names of places registered in the navigation system, or other related character strings associated with these.Then, the determination unit 18 determines whether the extracted related character strings supplied from the voice recognition unit 13 are included in the text data.

[0028] Then, the determination unit 18 calculates the relevance degree according to the determination result. In the first embodiment, when the related character string is included in the text data, the determination unit 18 sets the relevance degree to a threshold value or more. However, instead of this, the determination unit 18 may calculate the relevance degree according to the similarity between the text data and the related character string. In this case, the determination unit 18 may use an edit distance such as the Levenshtein distance or the Jaro-Winkler distance. The determination unit 18 may also calculate the relevance degree according to the number of hits in a predetermined time.

[0029] For example, assume that the user sets "Hase, Kamakura City, Kanagawa Prefecture" as the destination of the navigation device. In this case, the navigation information acquisition unit 16 acquires "Hase", which is the place name of the destination, as navigation information. The navigation information acquisition unit 16 may acquire the broader area "Kamakura" or "Kanagawa" as navigation information, or may acquire all of "Hase", "Kamakura" and "Kanagawa" as navigation information. The determination unit 18 refers to the related character string table 22 and extracts related character strings such as location names such as Kamakura and Kanagawa related to the place name "Hase" and famous names such as hydrangea and Great Buddha. After that, when the user utters "I'm looking forward to the hydrangea" or "Let's go to Kamakura!", the voice recognition unit 13 recognizes these voices and converts them into text data. The determination unit 18 determines that the text data includes "hydrangea" or "Kamakura", which are related character strings of the place name "Hase", and determines that the degree of relevance between the text data obtained by voice recognition of the utterance and the related character string of "Hase" is equal to or greater than a threshold value. Then, the record control unit 17 records in the image recording unit 21 the image data acquired by the image acquisition unit 11 before and after the utterance.

[0030] Also, assume that the user sets the destination to "Amusement Park E" from the search history of the navigation device. In this case, the navigation information acquisition unit 16 acquires the destination location name "Amusement Park E" as navigation information. The determination unit 18 refers to the related character string table 22 and extracts related character strings such as location names such as Maihama and Chiba, event names such as parade, and keyword "play" that are related to the location name "Amusement Park E". After that, when the user utters "Today, let's play!", the voice recognition unit 13 recognizes this voice and converts it into text data. The determination unit 18 determines that the text data includes the keyword "play", which is a related character string of the location name "Amusement Park E", and determines that the degree of association between the text data obtained by voice recognition of the utterance and the related character string of "Amusement Park E" is equal to or greater than a threshold value. The recording control unit 17 then records the image data acquired by the image acquisition unit 11 before and after the utterance in the image recording unit 21.

[0031] Also, suppose that the user sets a destination using the navigation registered location name "Grandma" registered in the navigation device corresponding to the location information of the user's grandmother's house. At this time, the navigation information acquisition unit 16 acquires the navigation registered location name "Grandma" as navigation information. The determination unit 18 refers to the related character string table 22 and extracts related character strings such as Kawagoe, which is a place name related to the navigation registered location name "Grandma", Saitama, which is a location name, and the keyword "homecoming". If the user then utters "I finally entered Saitama from Tokyo", the voice recognition unit 13 recognizes this voice and converts it into text data. The determination unit 18 determines that the text data includes the location name "Saitama", which is a related character string of the navigation registered location name "Grandma", and determines that the degree of association between the text data obtained by voice recognition of the utterance and the related character string of "Grandma" is equal to or greater than a threshold value. Then, the recording control unit 17 records the image data acquired by the image acquisition unit 11 before and after the utterance in the image recording unit 21.

[0032] In this way, by using the related string table 22, the determination unit 18 can easily calculate the degree of association.

[0033] The place name information of the departure point, destination, and intermediate points may be acquired by the navigation information acquisition unit 16 performing reverse geocoding from the latitude and longitude information.

[0034] FIG. 3 is a flowchart illustrating an example of a processing procedure of the image processing method according to the first embodiment.

[0035] First, in step S10, the image acquisition unit 11 of the image processing device 10 acquires image data from the camera 30, and the voice acquisition unit 12 acquires indoor voice data from the microphone 40. The image acquisition unit 11 supplies the image data to the recording control unit 17, and the voice acquisition unit 12 supplies the indoor voice data to the voice recognition unit 13.

[0036] Next, in step S11, the voice recognition unit 13 converts the indoor voice data into text data, separates it into words, and recognizes them. Then, the voice recognition unit 13 supplies the converted text data to the determination unit 18.

[0037] Next, in step S12, the navigation information acquisition unit 16 acquires navigation information from a navigation device. Note that the process shown in step S12 may be performed before step S10 or 11, or may be performed in parallel with step S10 or 11. The navigation information acquisition unit 16 supplies the acquired navigation information to the determination unit 18.

[0038] Next, in step S13, the determination unit 18 uses the related string table 22 to determine whether or not the text data includes a string related to the navigation information, and calculates the degree of relevance based on the determination result.

[0039] Next, in step S14, the determination unit 18 determines whether the degree of association is equal to or greater than a predetermined threshold, that is, whether an event has occurred. If the determination unit 18 determines that an event has occurred (Yes in step S14), the process proceeds to step S15, and if not (No in step S14), the process returns to step S10.

[0040] In step S15, the recording control unit 17 records the image data during the event occurrence period in the image recording unit 21. At this time, the recording control unit 17 may also record indoor sound data during the event occurrence period. Then, the recording control unit 17 ends the process.

[0041] Thus, according to the first embodiment, the image processing device 10 records image data when an event occurs in which a vehicle occupant utters a related character string such as a place name or a famous place name that is highly related to navigation information such as a destination or a current location. This allows the image processing device 10 to save a conversation scene related to a journey in the vehicle cabin as a digest image. Therefore, the user can conveniently look back on important scenes during the journey after the journey, etc.

[0042] <Embodiment 2> Next, a second embodiment of the present invention will be described with reference to Fig. 4. The second embodiment is characterized in that a combination of keywords included in a comment triggers recording of image data. The image processing device 10 according to the second embodiment has the same configuration and functions as the image processing device 10 according to the first embodiment, and therefore a description thereof will be omitted.

[0043] Fig. 4 is a flowchart showing an example of a processing procedure of an image processing method according to embodiment 2. The steps shown in Fig. 4 include steps S20 to S22 instead of steps S13 to S14 in Fig. 3. Note that the same steps as those in Fig. 3 are denoted by the same reference numerals and the description thereof will be omitted as appropriate.

[0044] In step S20, the determination unit 18 uses the related string table 22 in response to acquiring the navigation information and the text data to determine whether the related string is included in the text data and whether the related string is close to a predetermined keyword. Here, the predetermined keyword is preferably determined according to the type of the related string. Whether the related string is close to the predetermined keyword may be determined by determining whether the distance between these strings is less than a predetermined threshold. The distance between the strings may be calculated according to whether these strings are included in one sentence, or may be calculated based on the number of characters, the number of beats, or the number of phrases between the strings. For example, in the related string table 22 of FIG. 2, the predetermined keyword "play" is registered as a related string for the location name "E Amusement Park". At this time, in response to the passenger saying "Let's play at E Amusement Park," the determination unit 18 determines that the related string, the location name "E Amusement Park" and the keyword "play" are both included in the text data and are close to each other. Also, when the name of the navigation registered place of the destination is "Grandma", the predetermined keyword "Go home" is registered as the related character string, so in response to the occupant saying "Go home to Grandma's house", the determination unit 18 determines that the related character string, the navigation registered place name "Grandma", and the keyword "Go home", are both included in the text data and are close to each other. In the above example, the combination of the appearance of one related character string and the appearance of one keyword was the trigger, but there may be multiple related character strings or keywords that serve as the trigger. If the determination unit 18 determines that the related character string and the keyword are close to each other (Yes in step S20), the process proceeds to step S21, and if not (No in step S20), the process returns to step S10.

[0045] In step S21, the determination unit 18 calculates the relevance. In the second embodiment, when the related character string and the predetermined keyword are included in the text data in close proximity, the determination unit 18 determines the relevance to be equal to or greater than a threshold. However, instead of this, the determination unit 18 may calculate the relevance based on at least one of the similarity of the related character string, the distance between the related character string and the predetermined keyword, and the frequency of appearance of the above combination. Then, the determination unit 18 advances the process to step S22.

[0046] In step S22, the determination unit 18 determines whether the relevance is equal to or greater than a predetermined threshold, that is, whether an event has occurred. If the determination unit 18 determines that an event has occurred (Yes in step S22), the process proceeds to step S15, and if not (No in step S22), the process returns to step S10.

[0047] Thus, according to the second embodiment, the image processing device 10 records image data in response to a combination of a character string related to navigation information and a predetermined keyword. Therefore, scenes related to the trip, especially memorable scenes of the trip, can be determined with high accuracy and saved as digest images.

[0048] The image processing device 10 of the second embodiment can also provide the same effects as those of the first embodiment.

[0049] <Embodiment 3> Next, a third embodiment of the present invention will be described with reference to Figures 5 and 6. The third embodiment is characterized in that the determination that the scenery around the vehicle is related to the navigation information serves as a trigger for recording image data.

[0050] 5 is a block diagram showing an example of the configuration of an image processing system 1a according to embodiment 3. The image processing system 1a according to embodiment 3 basically has the same configuration and functions as the image processing system 1 according to embodiment 1. However, the image processing system 1a is different in that it includes an image processing device 10a instead of the image processing device 10.

[0051] The image processing device 10a has basically the same configuration and functions as the image processing device 10. However, the image processing device 10a has an image recognition unit 15 that recognizes the image acquired by the image acquisition unit 11, instead of the voice acquisition unit 12 and the voice recognition unit 13.

[0052] The image recognition unit 15 recognizes the subject or a character string included in the subject from the image data. The character string included in the subject may be a character string printed or drawn on the subject, or may be a character string represented by the shape of the subject. For example, the image recognition unit 15 recognizes the subject "Great Buddha" from an external image showing a Great Buddha. The image recognition unit 15 may recognize character strings "Great Buddha" or "Hydrangea" included in the subject from an external image showing a signboard as the subject. The image recognition unit 15 then generates text data including the character string representing the recognized subject. The image recognition unit 15 also recognizes the character string "Hase" included in a traffic sign or signboard from an image showing a traffic sign or signboard. The signboard may be a signboard showing a regional boundary such as a prefectural border. The image recognition unit 15 then generates text data including the character string representing the subject (such as "Great Buddha") or the recognized character string included in the subject. The image recognition unit 15 supplies the text data to the determination unit 18 of the recording control unit 17.

[0053] In the third embodiment, when image data and navigation information are related to each other, the recording control unit 17 determines that an event has occurred, and records the image data during the event occurrence period in the image recording unit 21. Here, the determination unit 18 of the recording control unit 17 calculates the degree of association between the character string recognized from the image and the navigation information, depending on whether or not the text data supplied from the image recognition unit 15 includes a related character string. Then, the determination unit 18 determines whether or not the degree of association is equal to or greater than a predetermined threshold, and determines that an event has occurred if the degree of association is equal to or greater than the predetermined threshold.

[0054] Fig. 6 is a flowchart showing an example of a processing procedure of an image processing method according to embodiment 3. The steps shown in Fig. 6 include steps S30 to S32 instead of steps S10, 11 and 15 shown in Fig. 3. Note that the same steps as those in Fig. 3 are denoted by the same reference numerals and the description thereof will be omitted as appropriate.

[0055] First, in step S30, the image acquisition unit 11 of the image processing device 10a acquires image data from the camera 30 that captures the scenery around the vehicle. The image acquisition unit 11 supplies the image data to the recording control unit 17 and the image recognition unit 15.

[0056] Next, in step S31, the image recognition unit 15 recognizes an object in the image or a character string included in the object. The image recognition unit 15 supplies the determination unit 18 with a character string representing the recognition result or text data including the character string of the recognition result.

[0057] Next, in steps S12 to 14, the image processing device 10a calculates the degree of association between the text data and the related character string of the navigation information, and determines whether or not an event has occurred based on the degree of association.

[0058] Next, in step S32, in response to determining that an event has occurred, the recording control unit 17 records image data during the event occurrence period in the image recording unit 21. Note that the image processing device 10a may have an audio acquisition unit 12, and in this case, the recording control unit 17 may record image data during the event occurrence period in association with indoor audio data during the event occurrence period. Then, the recording control unit 17 ends the process.

[0059] The image data used for image recognition in step S31 may be different from the image data recorded in step S32. For example, the former may be external image data of the scenery outside the vehicle, and the latter may be interior image data of the interior of the vehicle.

[0060] As described above, according to the third embodiment, the image processing device 10a records image data based on the relevance between the scenery around the vehicle and the navigation information, and therefore scenes related to the journey during travel can be saved as digest images. Therefore, the user can conveniently look back on important scenes during travel after the journey. Note that the image processing device 10a uses the image recognition result to determine whether the image is related to the journey, and therefore can easily calculate the relevance.

[0061] <Embodiment 4> Next, a fourth embodiment of the present invention will be described with reference to Fig. 7. The fourth embodiment is characterized in that a combination of character strings included in an image triggers recording of image data. An image processing device 10a according to the fourth embodiment has the same configuration and functions as the image processing device 10a according to the third embodiment, and therefore a description thereof will be omitted.

[0062] Fig. 7 is a flowchart showing an example of a processing procedure of an image processing method according to embodiment 4. The steps shown in Fig. 7 include steps S40 to S42 instead of steps S13 to S14 in Fig. 6. Note that the same steps as those in Fig. 6 are denoted by the same reference numerals and the description thereof will be omitted as appropriate.

[0063] In step S40, the determination unit 18 determines whether or not the character string included in the subject of the external image includes a related character string of the navigation information and a character string indicating a predetermined content in response to acquiring the navigation information from the navigation information acquisition unit 16. For example, if the related character string is the character string "Hase" indicating the name of the destination, the character string indicating the predetermined content may be a character string "Xkm" indicating distance information to the destination or a character string "Y minutes traffic jam" indicating traffic information to the destination. The determination unit 18 may determine that the two character strings are included in response to detecting the two character strings at a position less than a predetermined distance in the external image. In this case, the distance between the two character strings may be the number of pixels between the two character strings in the external image normalized with respect to the number of pixels occupied by one character or the number of pixels of the subject. In the above example, the combination of one related character string and one character string indicating the predetermined content became a trigger, but the related character string or the character string indicating the predetermined content that becomes a trigger may be multiple. If the judgment unit 18 determines that the text data including the character string contained in the subject recognized by the image recognition unit 15 includes the above combination of related character strings (Yes in step S40), the judgment unit 18 proceeds to step S41; if not (No in step S40), the judgment unit 18 returns to step S30.

[0064] In step S41, the determination unit 18 calculates the relevance. In the fourth embodiment, when text data including a string included in a subject of an external image includes a related string and a string indicating a predetermined content, the determination unit 18 sets the relevance to a threshold value or more. However, instead of this, the determination unit 18 may calculate the relevance according to at least one of the similarity of the related string, the distance between the related string and a string indicating the predetermined content, and the frequency of appearance of the above combination in a predetermined number of frames. Then, the determination unit 18 advances the process to step S42.

[0065] In step S42, the determination unit 18 determines whether the degree of association is equal to or greater than a predetermined threshold, that is, whether an event has occurred. If the determination unit 18 determines that an event has occurred (Yes in step S42), the process proceeds to step S32, and if not (No in step S42), the process returns to step S30.

[0066] According to the fourth embodiment, the image processing device 10a records image data when a combination of a character string related to navigation information and a character string indicating a predetermined content appears in an external image. Therefore, the image processing device 10a can recognize a scene related to a journey with high accuracy.

[0067] The image processing device 10a of the fourth embodiment can also achieve the same effects as those of the third embodiment.

[0068] <Embodiment 5> Next, a fifth embodiment of the present invention will be described with reference to Figures 8 to 12. The fifth embodiment is characterized in that image processing such as adding subtitles to an image is performed.

[0069] 8 is a block diagram showing an example of the configuration of an image processing system 1b according to embodiment 5. The image processing system 1b according to embodiment 5 basically has the same configuration and functions as the image processing system 1 according to embodiment 1. However, the image processing system 1b is different in that it includes an image processing device 10b instead of the image processing device 10.

[0070] The image processing device 10b has basically the same configuration and functions as the image processing device 10. However, the image processing device 10b differs in that it has a recording control unit 17b and a determination unit 18b instead of the recording control unit 17 and the determination unit 18, and that the memory unit 20b stores an association mode table 23 and an image processing mode table 24.

[0071] The recording control unit 17b includes a determination unit 18b and an image processing unit 19. As a modification of the fifth embodiment, the recording control unit 17b may be configured to include the determination unit 18 and the image processing unit 19, but not include the associated mode table 23 and the image processing mode table 24. In other words, the determination unit 18 in the modification of the fifth embodiment may perform the same determination as in the above-mentioned embodiments 1-4, and may set the image processing mode to a predetermined mode without identifying the associated mode or determining the image processing mode, which will be described later.

[0072] In addition to the functions of the determination unit 18, the determination unit 18b has a function of identifying an association aspect that indicates how an association character string is associated with navigation information when the association character string is included in the text data converted by the voice recognition unit 13. The determination unit 18b may use an association aspect table 23 when identifying the association aspect.

[0073] The associated aspect table 23 is a table that determines associated aspect identification information (associated aspect ID) according to a combination of the type of navigation information and the type of associated character string.

[0074] When the determination unit 18b determines that an event has occurred, the image processing unit 19 performs image processing on the image of the image data during the event occurrence period. At this time, the image processing unit 19 generates superimposed image data in which at least one of an image based on the text data converted by the voice recognition unit 13 and an image related to the event is superimposed. The image based on the text data is, for example, a subtitle image in which the text data is visualized as a character string using a predetermined font. For example, when the text data is in Japanese, the image based on the text data may be converted into a subtitle image after being converted into kanji or katakana. The image based on the text data may be an extracted word included in the text data, or may take a form in which a part of the text data is omitted or predetermined text data is added. The image based on the text data may be a mark image previously associated with the text data by a database not shown. The image related to the event may be an image related to a related character string. For example, when the related character string is "Great Buddha", it may be an illustrated image of a Great Buddha.

[0075] The recording control unit 17b stores the superimposed image data generated by the image processing unit 19 in the image recording unit 21. At this time, the recording control unit 17b preferably records the data in the image recording unit 21 in a recording format that prohibits overwriting. If the image processing device 10b has a recording mode for abnormality recording, it is preferable that the recording control unit 17b does not perform the above-mentioned image processing on images recorded in the recording mode for abnormality recording. This makes it possible to easily view the recording file after movement, distinguishing it from data recorded in the above-mentioned normal recording mode.

[0076] In the fifth embodiment, the image processing unit 19 determines the image processing mode according to the associated mode when performing the above image processing. The image processing unit 19 may use an image processing mode table 24 to determine the image processing mode.

[0077] The image processing mode table 24 is a table that associates the related mode with the image processing mode. In the fifth embodiment, the related mode ID is used as the related mode.

[0078] In this way, the image processing unit 19 generates superimposed image data by superimposing at least one of an image based on the text data and an image related to the event on an image during the event occurrence period, and stores the superimposed image data in the image recording unit 21, thereby allowing the user to look back on memorable scenes from the trip in easy-to-understand digest images as a record of the trip.

[0079] The image processor 19 may determine the manner of the image to be superimposed, particularly the manner of the image based on the text data to be superimposed, based on the calculated relevance. This allows image presentation, particularly subtitle presentation, to be performed according to the degree of relevance of the passenger's remark.

[0080] 9 is a diagram showing an example of a data structure of the related aspect table 23 according to the fifth embodiment. The related aspect ID is determined based on a combination of the type of navigation information and the type of related character string. The related aspect ID may also be determined based on a combination of the type of navigation information, the type of related character string, and the appearance of a predetermined additional keyword.

[0081] FIG. 10 is a diagram showing an example of the data structure of the image processing mode table 24 according to the fifth embodiment. The image processing mode table 24 shown in this figure associates an associated mode ID with an image processing mode. The image processing mode includes matters related to the display mode of the superimposed subtitles and other image processing modes. In this figure, matters related to the mode of the superimposed subtitles are the character size, font, and subtitle superimposition position of the subtitles, but may also include color. Other image processing modes include designation of subtitle content according to an event (here, related character strings), addition of an image related to the event, solid black processing, addition of a mark image, addition of a balloon image surrounding the subtitles, color change, monochrome processing of a color image, black and white inversion processing, and mosaic processing. The character size in FIG. 10 is shown as an example in five stages from 5 (maximum) to 1 (minimum), but may be ten stages or may be designated by the actual character size (for example, the number of pixels of the font height).

[0082] For example, when the determination unit 18b determines that the text data of the voice recognized by the voice recognition unit 13 includes the destination name of the navigation information (i.e., the place name of the destination) or the location name of the destination, the determination unit 18b uses the associated mode table 23 to identify the associated mode as associated mode ID "1". The image processing unit 19 uses the image processing mode table 24 to extract the character size, font, and subtitle superimposition position of the subtitle having the associated mode ID "1" as "5", "Gothic", and "bottom of screen", respectively. The image processing unit 19 may add a predetermined character to the text data based on the associated mode, may select a mark image, or may determine the size and color of the mark image, using an image processing mode table not shown.

[0083] Furthermore, when the determination unit 18b determines that the text data of the voice recognized by the voice recognition unit 13 includes the additional keyword "traffic jam" in addition to the name of the route point or the location name of the route point in the navigation information, the determination unit 18b identifies the related aspect as related aspect ID "4" using the related aspect table 23. The image processing unit 19 uses the image processing aspect table 24 to extract the font size, font, and subtitle superimposition position of the subtitle with related aspect ID "4" as "4", "extra bold gothic", and "center of screen", respectively.

[0084] Fig. 11 is a flowchart showing an example of a processing procedure of an image processing method according to embodiment 5. The steps shown in Fig. 11 include steps S50 to S52 instead of step S15 in Fig. 3 according to embodiment 1. Note that the same steps as those in Fig. 3 are denoted by the same reference numerals and the description thereof will be omitted as appropriate.

[0085] In step S50, in response to determining that an event has occurred (Yes in step S14), the determination unit 18b of the image processing device 10b identifies the related aspect of the related character string by using the related aspect table 23. Then, the determination unit 18b supplies the identified related aspect ID to the image processing unit 19.

[0086] Next, in step S51, the image processing unit 19 refers to the image processing mode table 24, and executes image processing to generate a subtitle image by visualizing the text data in an image processing mode corresponding to the related mode ID, and superimposes the subtitle image on the image acquired by the image acquisition unit 11. As a result, the image processing unit 19 generates superimposed image data in which the subtitle image generated in the mode corresponding to the related mode ID or another image is superimposed.

[0087] Next, in step S52, the recording control unit 17b records the image data that has been subjected to image processing in the image recording unit 21. At this time, the recording control unit 17b may record the image data to be recorded in association with the corresponding indoor audio data. Then, the recording control unit 17b ends the process.

[0088] The recording control unit 17b may be included as a component of the image processing device 10, 10a according to the second to fourth embodiments in place of the recording control unit 17. In this case, the above-mentioned steps S50 to S52 are included instead of step S15 in Fig. 4 according to the second embodiment or step S32 in Figs. 6 to 7 according to the third and fourth embodiments.

[0089] Thus, according to the fifth embodiment, the image processing device 10b saves impressive scenes of the journey as digest images with subtitles. Therefore, after the journey, the passengers can easily look back on and enjoy the images saved as a record of the journey, just like watching a variety show.

[0090] The image processing device 10b of the fifth embodiment can also achieve the same effects as those of the first embodiment.

[0091] <Other embodiments> The present invention is not limited to the first to fifth embodiments, and can be modified as appropriate without departing from the spirit and scope of the present invention. For example, the image processing device 10 of the first embodiment may include the image recognition unit 15 of the third or fourth embodiment. In this case, the determination unit 18 of the recording control unit 17 may calculate the relevance based on the text data supplied from the image recognition unit 15 instead of or in addition to the text data supplied from the voice recognition unit 13. Note that the present invention may be applied to the image processing device 10 of the second embodiment or the image processing device 10b of the fifth embodiment instead of the image processing device 10 of the first embodiment. This makes it possible to determine whether an image is related to a trip from various angles, thereby preventing a scene that should be saved as a record of the trip from being missed, and also makes it possible to more accurately determine whether an image should be saved as a record of the trip.

[0092] In the above embodiment, the present invention has been described as being configured as hardware, but the present invention is not limited to this. Any process of the present invention can also be realized by causing a processor to execute a computer program.

[0093] In the above example, the program can be stored and provided to the computer using various types of non-transitory computer readable media. The non-transitory computer readable media includes various types of tangible storage media. Examples of the non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, DVD (Digital Versatile Disc), BD (Blu-ray (registered trademark) Disc), and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). The program may also be provided to the computer by various types of transitory computer readable media. Examples of the transitory computer readable media include electric signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire or an optical fiber, or via a wireless communication path.

[0094] In the above-mentioned embodiment, the computer is configured as a computer system including a personal computer, a word processor, etc. However, the present invention is not limited to this, and the computer can also be configured as a server of a LAN (local area network), a host for computer (personal computer) communication, a computer system connected to the Internet, etc. It is also possible to distribute functions to each device on the network and configure a computer as a whole network.

[0095] The order of execution of each process in the system and method shown in the claims, specification, and drawings is not specifically stated as "before," "prior to," etc., and may be implemented in any order as long as the output of a previous process is not used in a subsequent process. Even if the operational flow in the claims, specification, and drawings is described using "first," "next," etc. for convenience, it does not mean that it is essential to implement the process in this order.

[0096] Furthermore, some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. (Appendix 1) an image acquisition unit that acquires image data of a range including the surroundings of the vehicle; A voice acquisition unit that acquires indoor voice data in the vehicle; a navigation information acquisition unit for acquiring navigation information of the vehicle; a recording control unit that, when an event occurs in which it is determined that the degree of association between at least one of the image data and the indoor voice data and the navigation information is equal to or greater than a predetermined threshold, records the image data for a period including a period in which the event occurs in an image recording unit; An image processing device comprising: (Appendix 2) a voice recognition unit for converting the indoor voice data into text data; an image recognition unit that recognizes an object or a character string included in the object from an image of the image data, and generates text data of the recognized character string representing the object or the character string included in the object; The recording control unit has a determination unit that calculates the relevance level depending on whether the text data includes a character string related to the navigation information. 2. An image processing device as described in claim 1. (Appendix 3) The determination unit calculates the degree of association in response to determining that the text data includes a character string related to the navigation information and a predetermined keyword in close proximity to each other. 3. An image processing device as described in claim 2. (Appendix 4) The determination unit calculates the relevance level in response to determining that a character string included in the subject includes a character string related to the navigation information and a character string indicating a predetermined content. 3. An image processing device as described in claim 2. (Appendix 5) the character string related to the navigation information indicates at least one of a keyword related to a name of a location, a place name where the location is located, a local specialty related to the location, a famous place or facility related to the location, an event related to the location, traffic congestion information, a distance between a current position and the location, and a usage purpose of a facility related to the location; The point indicates a starting point, a destination, a waypoint, a current location, a registered location, or a destination or waypoint in a travel history. 5. An image processing device according to any one of claims 2 to 4. (Appendix 6) the recording control unit further includes an image processing unit that generates superimposed image data by superimposing at least one of an image based on the text data and an image related to the event on the image data; The recording control unit records, in the image recording unit, the superimposed image data for a period including a period during which the event occurs, when the event occurs. 3. An image processing device as described in claim 2. (Appendix 7) The image processing unit determines an image processing mode of at least one of an image based on the text data and an image related to the event, based on an association mode between the text data and a character string related to the navigation information. 7. The image processing device according to claim 6. (Appendix 8) acquiring image data of a range including an area surrounding the vehicle; acquiring indoor audio data within a vehicle interior; obtaining navigation information for the vehicle; when an event occurs in which it is determined that the degree of association between at least one of the image data and the indoor voice data and the navigation information is equal to or greater than a predetermined threshold, recording the image data for a period including a period in which the event occurs in an image recording unit; An image processing method comprising: (Appendix 9) A process of acquiring image data of a range including the surroundings of the vehicle; A process of acquiring indoor voice data in the interior of the vehicle; obtaining navigation information for the vehicle; a process of recording, in a case where an event occurs in which it is determined that the degree of association between at least one of the image data and the indoor voice data and the navigation information is equal to or greater than a predetermined threshold, the image data for a period including a period during which the event occurs in an image recording unit; An image processing program that allows a computer to execute the above. [Explanation of symbols]

[0097] 1,1a,1b Image processing system 10, 10a, 10b Image processing device 11 Image acquisition section 12 Voice Acquisition Unit 13 Voice Recognition Unit 15 Image Recognition Unit 16 Navigation information acquisition section 17, 17b Recording control section 18,18b Judgment part 19 Image processing section 20,20b Storage section 21 Image recording unit 22 Related String Table 23 Related aspect table 24 Image processing status table 30 Camera 40. Mike

Claims

1. an image acquisition unit that acquires image data of a range including the surroundings of the vehicle; A voice acquisition unit that acquires indoor voice data in the vehicle; a navigation information acquisition unit for acquiring navigation information of the vehicle; a recording control unit that, when an event occurs in which it is determined that the degree of association between at least one of the image data and the indoor voice data and a related character string related to the navigation information is equal to or greater than a predetermined threshold, records the image data for a period including a period in which the event occurs in an image recording unit; An image processing device comprising:

2. a voice recognition unit for converting the indoor voice data into text data; an image recognition unit that recognizes an object or a character string included in the object from an image of the image data, and generates text data of the recognized character string representing the object or the character string included in the object; The recording control unit includes a determination unit that calculates the degree of association between the text data and a character string related to the navigation information by using at least one of a Levenshtein distance and a Jaro-Winkler distance. The image processing device according to claim 1 .

3. The related character string is identified by referring to a related character string table which stores, in association with the name of the location indicated by the navigation information, the name of the location and the name of the area where the location is located, the names of famous places, famous places, and famous facilities related to the location, events, the nearest station, and keywords corresponding to the intended use of the facilities related to the location, and the registered location names indicating the previously registered locations or facilities.

3. The image processing device according to claim 1 or 2.

4. At least one of an image acquisition unit that acquires image data of a range including the periphery of a vehicle, and a voice acquisition unit that acquires indoor voice data of the interior of the vehicle; a navigation information acquisition unit for acquiring navigation information of the vehicle; a recording control unit that, when an event occurs for which it is determined that the degree of association between at least one of the image data and the indoor voice data and the navigation information is equal to or greater than a predetermined threshold, generates superimposed image data by superimposing, on the image data, at least one of an image based on at least one of the image data and the indoor voice data and an image related to the event, and records, in an image recording unit, the superimposed image data for a period including a period during which the event occurs; An image processing device comprising:

5. acquiring image data of a range including an area surrounding the vehicle; acquiring indoor audio data within a vehicle interior; obtaining navigation information for the vehicle; a step of recording, in an image recording unit, the image data for a period including a period during which the event occurs, when it is determined that a degree of association between at least one of the image data and the indoor voice data and a related character string related to the navigation information is equal to or greater than a predetermined threshold value. An image processing method comprising:

Images