Image generation apparatus, image generation method, and storage medium of image generation program

Abstract

Provided is an image generation device (1) including: an in-vehicle interior camera (14a) that captures an in-vehicle interior including an occupant; an outside camera (14b) that captures surroundings of the host vehicle; an image acquisition unit (111) that acquires an appearance image of the host vehicle; and an image generation unit (113) that generates a recording image by combining at least one of an in-vehicle interior image captured by the in-vehicle interior camera (14a) and an outside image captured by the outside camera (14b) with the appearance image acquired by the image acquisition unit (111).

Description

Technical Field

[0001] This invention relates to an image generation apparatus, an image generation method, and an image generation program. Background Technology

[0002] Previously, there were known devices that generated a composite image of a subject inside a vehicle with the exterior scenery as the background, based on images taken by a vehicle-mounted camera that photographs a subject inside a vehicle and images taken by a vehicle-mounted camera that photographs the scenery outside the vehicle (see, for example, Patent Document 1).

[0003] However, if, as described in Patent Document 1 above, the image of the scenery outside the vehicle is only used to photograph the subject inside the vehicle, the resulting composite image will be far from the actual situation and is not recommended for recording purposes.

[0004] Existing technical documents

[0005] Patent documents Patent document 1: Japanese Patent Application Publication No. 2020-095406 (JP2020-095406A). Summary of the Invention

[0006] An image generation apparatus according to a technical solution of the present invention comprises: a first shooting device for shooting an image of a vehicle interior including occupants; a second shooting device for shooting an image of the vehicle's surroundings; an acquisition unit for acquiring an image of the vehicle's exterior; and a generation unit for combining at least one of the interior image acquired by the first shooting device and the exterior image acquired by the second shooting device with the exterior image acquired by the acquisition unit to generate a recording image.

[0007] Another technical solution of the present invention provides an image generation method comprising the following steps: acquiring an interior image of a vehicle from a first shooting device, which captures images of the interior of the vehicle including the occupants; acquiring an exterior image of the vehicle from a second shooting device, which captures images of the area around the vehicle; acquiring an exterior image of the vehicle stored in a storage unit; and combining at least one of the interior image and the exterior image with the exterior image to generate a recording image.

[0008] Another technical solution of the present invention uses a storage medium that can be read by a computer and is used to store an image generation program. The image generation program causes the computer to perform the following steps: acquiring an interior image of the vehicle obtained by the first shooting device, which captures images of the vehicle interior including the occupants; acquiring an exterior image of the vehicle obtained by the second shooting device, which captures images of the vehicle's surroundings; acquiring an exterior image of the vehicle stored in the storage unit; and combining at least one of the interior image and the exterior image with the exterior image to generate a recording image. Attached Figure Description

[0009] The objectives, features, and advantages of the present invention are further illustrated by the following description of embodiments in conjunction with the accompanying drawings.

[0010] Figure 1 This is a block diagram that schematically illustrates the overall structure of a system incorporating an image generation apparatus according to an embodiment of the present invention; Figure 2 This is an example Figure 1 A block diagram of the main structural components of an image generation device; Figure 3 This is a diagram showing an example of an image used for recording; Figure 4 This is a diagram showing another example of an image used for recording; Figure 5 This is another example of an image used for recording; Figure 6 This is another example of an image used for recording; Figure 7 This is an example Figure 1 A block diagram of the main structural components of a user terminal; Figure 8 It is shown by Figure 2 A flowchart of an example of the processing performed by the controller. Detailed Implementation

[0011] Hereinafter, embodiments of the invention will be described with reference to the accompanying drawings. The image generation apparatus of the present invention is an apparatus for generating recording images for occupants to review memories of vehicle trips or drives, based on image data acquired by an in-vehicle camera. It should be noted that, sometimes, to distinguish it from other vehicles, the vehicle to which the image generation apparatus of this embodiment is applied is referred to as "this vehicle". Figure 1 This is a schematic block diagram showing the overall structure of the system 100 including the image generation apparatus 1 of this embodiment. Figure 1 As shown, system 100 includes image generation device 1 and user terminal 2.

[0012] The image generating device 1 and the user terminal 2 can be communicatively connected via the communication network 3. The communication network 3 includes not only public radio communication networks such as the Internet and mobile phone networks, but also closed communication networks set up for each designated management area, such as wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] User terminal 2 is a communication terminal such as a smartphone, used by passengers or other users of the vehicle. It should be noted that... Figure 1 The example shows a user terminal 2, but the system 100 also includes user terminals 2 corresponding to each user.

[0014] Figure 2 This is an example Figure 1 A block diagram of the main structure of the image generating apparatus 1. The image generating apparatus 1 includes a controller 10, a communication unit 13, an imaging device 14, a microphone (hereinafter referred to as a microphone) 15, a display unit 16, and an operation unit 17. The communication unit 13 is a communication interface that connects the image generating apparatus 1 to the communication network 3. The image generating apparatus 1 transmits and receives information via the communication unit 13 and a user terminal 2 connected to the communication network 3.

[0015] The imaging device 14 includes a camera 14a for photographing the interior of the vehicle (hereinafter referred to as the interior camera) and a camera 14b for photographing the surroundings of the vehicle (hereinafter referred to as the exterior camera). Cameras 14a and 14b each have imaging elements such as a CCD (charge-coupled device) and a CMOS (complementary metal-oxide-semiconductor). The image data acquired by cameras 14a and 14b is output to the processing unit 11. A microphone 15 is used to input sound from inside the vehicle as an audio signal. The audio signal input by the microphone 15 is output to the processing unit 11 as audio data via an A / D converter (not shown).

[0016] The display unit 16 is composed of a display device such as a liquid crystal display or an organic EL display equipped with a touch panel. The operation unit 17 is composed of the touch panel of the display unit 16 and can be used to input various instructions from the user. It should be noted that the operation unit 17 can also be composed of input devices such as switches or buttons.

[0017] The controller 10 is configured as a computer having an arithmetic unit 11 such as a CPU (microprocessor), a storage unit 12 such as a ROM (read-only memory) and RAM (random access memory), and other peripheral circuits (not shown) such as I / O interfaces. The arithmetic unit 11 functions as an image acquisition unit 111, a detection unit 112, an image generation unit 113, and an output unit 114 by executing programs stored in the storage unit 12.

[0018] The image acquisition unit 111 acquires image data showing the exterior of the vehicle (hereinafter referred to as exterior image data or simply exterior image). More specifically, the image acquisition unit 111 reads and acquires the exterior image stored in the storage unit 12.

[0019] It should be noted that, according to the registration instruction described later, the appearance image is stored in the storage unit 12 in advance. Specifically, when the image acquisition unit 111 receives the registration instruction from the user terminal 2, it stores the appearance image received together with the registration instruction in the storage unit 12.

[0020] In addition, the image acquisition unit 111 acquires image data captured by the in-vehicle camera 14a (hereinafter referred to as in-vehicle image data or simply in-vehicle image). Furthermore, the image acquisition unit 111 acquires image data captured by the out-of-vehicle camera 14b (hereinafter referred to as out-of-vehicle image data or simply out-of-vehicle image).

[0021] The detection unit 112 detects specific locations (specific landmarks or specific areas) existing around the vehicle based on images outside the vehicle. Specifically, the detection unit 112 uses a multimodal LLM (Large Language Model) to detect specific locations using images outside the vehicle as input. Specific landmarks include historical sites, distinctive buildings or natural landscapes, famous attractions, signs, etc. Specific areas include administrative boundaries, stunning scenery, etc. It should be noted that the detection unit 112 can also detect specific locations existing around the vehicle based on map information stored in the storage unit 12 and the current location of the vehicle detected by onboard sensors (positioning sensors) not shown.

[0022] The detection unit 112 also detects specific interior conditions based on the interior images. Specific interior conditions include an occupant's heightened emotions due to a particular topic, or surprise at the view seen through the car window. The detection unit 112 uses a machine learning model—more specifically, an emotion classification model—to identify emotions based on the occupant's facial expressions, taking the interior images as input, and detects specific interior conditions based on the identification results.

[0023] The emotion recognition results based on the emotion classification model include emotion scores representing various emotions of the occupant (excitement, surprise, happiness, anger, sadness, etc.) using values ​​from 0.0 (none) to 1.0 (maximum). When the emotion score corresponding to a specified emotion (e.g., excitement or surprise) exceeds a predetermined threshold, the detection unit 112 determines that the occupant is in a state of heightened emotion or surprise. In this way, the detection unit 112 detects specific conditions within the vehicle interior.

[0024] It should be noted that the detection unit 112 can also detect specific in-vehicle interior conditions based on audio data acquired by the microphone 15 (hereinafter referred to as in-vehicle interior audio data or simply in-vehicle interior sound). In this case, the detection unit 112 uses an emotion classification model, taking the in-vehicle interior sound as input, to identify emotions based on the occupant's voice and speaking volume. Additionally, the detection unit 112 can also detect specific in-vehicle interior conditions based on in-vehicle interior images and in-vehicle interior sound.

[0025] The detection unit 112 also detects the occupant's gaze direction based on the in-vehicle interior image. Specifically, the detection unit 112 detects the occupant's facial orientation, eye position, pupil movement, etc., contained in the in-vehicle interior image, and infers the occupant's gaze direction based on these detection results. If the inferred gaze direction does not change within a specified time, the detection unit 112 detects that gaze direction as the gaze direction.

[0026] It should be noted that the detection unit 112 can also detect the gaze direction based on the occupant's gestures (pointing gestures, etc.) recognized from the in-vehicle interior image. Furthermore, the detection unit 112 can also identify the gaze direction based on the occupant's conversation content recognized from the in-vehicle interior sound, especially when the conversation contains phrases indicating direction such as "forward," "right," "south," or "east." In addition, the detection unit 112 can use a machine learning model, more specifically a gaze estimation model, to estimate the gaze using the in-vehicle interior image as input, and detect the gaze direction based on the estimation result.

[0027] The image generation unit 113 uses multimodal LLM to combine at least one of the interior and exterior images with the exterior image to generate recording image data (hereinafter referred to as the recording image). It should be noted that when the image acquisition unit 111 acquires multiple exterior images from the storage unit 12, multiple exterior images can be used in the generation of the recording image data. Here, the generation of the recording image by the image generation unit 113 will be described.

[0028] When a specific interior condition is detected by the detection unit 112, the image generation unit 113 generates a recording image representing the specific interior condition based on the interior image. At this time, the image generation unit 113 generates prompts (instruction statements) for input to the multimodal LLM in such a way that the recording image at least includes an occupant who is emotionally agitated or surprised.

[0029] Examples of prompts could include "Generate an image representing the situation inside the vehicle, including occupants who are excited or surprised." The image generation unit 113 inputs the generated prompt and the image of the vehicle interior together into a multimodal LLM. Thus, using the multimodal LLM, a recording image representing a specific situation inside the vehicle interior is generated.

[0030] Figure 3 This is a diagram showing an example of an image used for recording. Figure 3 The image shown represents a specific situation inside a train car, specifically an image depicting a passenger's heightened emotions due to a friend's romantic conversation. It should be noted that, as... Figure 3 As shown, the images used for recording are preferably generated in a manner that includes all occupants.

[0031] On the other hand, when a specific location is detected by the detection unit 112, the image generation unit 113 generates a recording image containing the vehicle and the specific location based on the exterior image and the external image. At this time, the image generation unit 113 generates prompts for input to the multimodal LLM to generate a recording image of the vehicle with the specific location as the background.

[0032] Examples of prompts could include "Generate an image of this vehicle with a specific landmark or area as the background." The image generation unit 113 inputs the generated prompt, the exterior image, and the exterior view image into the multimodal LLM. Thus, if the exterior image includes landmarks such as Mount Fuji or Tokyo Tower, the multimodal LLM generates a recording image of the vehicle with those landmarks as the background.

[0033] Figure 4 This is a diagram showing another example of an image used for recording. Figure 4 The image shown is a recording taken against a backdrop of a specific location (Mount Fuji) visible from behind the vehicle. Normally, even when riding in a stylish or favorite car, occupants cannot take photos of a moving vehicle from outside the car and create a record. However, as... Figure 4 As shown, by generating recording images that include the exterior of the vehicle (Vehicle VH1), it is possible to provide the user with images of the vehicle they are driving or riding in, taken from outside the vehicle. Additionally, as... Figure 4 As shown, by generating recording images that include the occupants, i.e., the users (DR), it is possible to provide images that are closer to the actual situation.

[0034] When the detection unit 112 detects both a specific interior condition and a specific location, the image generation unit 113 generates a recording image that includes the occupants, the vehicle itself, and the specific location, based on the interior image, exterior image, and external image. At this time, if the detection unit 112 detects the occupants' gaze direction, the image generation unit 113 generates a recording image that takes into account the gaze direction and the camera angle. Specifically, the image generation unit 113 generates prompts for input to the multimodal LLM to generate a recording image that takes into account the gaze direction and the camera angle.

[0035] Examples of prompts could include phrases like, "Generate an image of this vehicle with a specific landmark or area as a background, taken from the opposite side of the gaze direction at a camera angle facing the gaze direction." The image generation unit 113 inputs the generated prompt, along with information indicating the gaze direction, an exterior image, and an exterior image, into a multimodal LLM. Thus, the multimodal LLM generates the image described later. Figure 5 The image shown is a record.

[0036] Figure 5This is another example of an image used for recording. Figure 5 The image shown is a recording image generated when the occupants are looking at a specific location (the magnificent mountain MT) visible on the right side of the vehicle. Specifically, it shows a recording image of the vehicle (vehicle VH2) taken from the opposite side (i.e., the left side of the vehicle) from the direction of gaze (i.e., the direction where the mountain MT is located), at a camera angle facing the direction of gaze, with the mountain MT as the background.

[0037] It should be noted that, as Figure 5 As shown in the example, the image generation unit 113 can also add an additional instruction to the prompts used for inputting to the multimodal LLM, to remove a portion of the vehicle body that obscures the occupants when viewed from the perspective of the recorded image (in... Figure 5 In the example, this refers to the right side and front right section of the vehicle body. As an example of additional instructions, one could say, "Treat the vehicle as a sectional model so that the occupants inside the vehicle are visible." This allows users to easily review the recorded images to determine the interior conditions at a specific location.

[0038] In addition, the image generation unit 113 can also generate prompts for input to the multimodal LLM to generate recording images containing specific locations visible from inside the vehicle through the front window, rear window, or side window, and the occupants inside the vehicle.

[0039] Examples of prompts could include "generating an image that includes a specific landmark or area visible from inside the vehicle through the window, and occupants looking at the specific landmark or area." The image generation unit 113 inputs the generated prompt, along with the interior and exterior images, into a multimodal LLM. Thus, through the multimodal LLM, the image described later is generated. Figure 6 The image shown is a record.

[0040] Figure 6 This is another example of an image used for recording. Figure 6 The text shows that it contains information from... Figure 5 The recorded images include a specific location (mountain MT) visible through the right-side window inside the VH2 vehicle's interior, and images of occupants looking towards that location. (The images are captured using methods such as...) Figure 6 The recorded images shown allow the user to review the interior conditions at the detected specific location. The image generation unit 113 can also generate... Figure 5 Such records use images and Figure 6 Such records use both images.

[0041] The output unit 114 outputs the recording image generated by the image generation unit 113 to the storage unit 12. Thus, the recording image is stored in the storage unit 12. It should be noted that the output unit 114 can also output the recording image to an external device (server device, storage device) via the communication unit 13. Furthermore, upon receiving a transmission command (described later) from the user terminal 2, the output unit 114 reads the recording image from the storage unit 12 and outputs (transmits) it to the user terminal 2.

[0042] Figure 7 This is an example Figure 1 The block diagram shows the main structural components of user terminal 2. User terminal 2 includes a controller 20, a communication unit 23, a display unit 24, an operation unit 25, a camera unit 26, a microphone 27, and a speaker 28. The storage unit 22 stores various control programs, thresholds used in the programs, and other information.

[0043] The communication unit 23 is a communication interface that connects the user terminal 2 to the communication network 3. The display unit 24 is composed of a display device such as a liquid crystal display or an organic EL display with a touch panel. The operation unit 25 is composed of the touch panel of the display unit 24 and can be used to input various instructions from the user.

[0044] The imaging unit 26 comprises a camera with imaging elements such as CCD and CMOS. Image data acquired by the imaging unit 26 is output to the processing unit 21. Microphone 27 is used to input ambient sound as an audio signal. The audio signal input by microphone 27 is output as audio data to the processing unit 21 via an A / D converter (not shown). Speaker 28 outputs the audio data from the processing unit 21 as an audio signal via a D / A converter (not shown).

[0045] The controller 20 is configured as a computer having an arithmetic unit 21 such as a CPU (microprocessor), a storage unit 22 such as ROM and RAM, and other peripheral circuits not shown, such as I / O interfaces. The arithmetic unit 21 functions as an image registration unit 211 and a display control unit 212 by executing programs stored in the storage unit 22.

[0046] The image registration unit 211 receives a registration request from the user for an image of the vehicle's exterior. Specifically, the image registration unit 211 first generates image information for accepting the registration request for the exterior image and outputs this image information to the display unit 24. As a result, a registration request acceptance screen (not shown) is displayed on the display unit 24. It should be noted that the image registration unit 211 can output the aforementioned image information to the display unit 24 when the specified application installed on the user terminal 2 is started, or it can output it at other times.

[0047] The user operates the camera unit 26 via the operation unit 25 to take pictures of the vehicle's exterior. The captured image (exterior image) is then stored in the storage unit 22 of the user terminal 2. The user then uses the operation unit 25 to access the registration request acceptance screen, selects the vehicle's exterior image stored in the storage unit 22, and requests to register the exterior image.

[0048] When the image registration unit 211 receives a registration request for an appearance image from the user, it reads the appearance image selected by the user from the storage unit 22 and sends the appearance image and registration instruction to the image generation device 1 via the communication unit 23.

[0049] The display control unit 212 receives a display request for a recorded image from the user. Specifically, the image registration unit 211 first generates image information for accepting the display request for the recorded image and outputs this image information to the display unit 24. As a result, a request acceptance screen (not shown) is displayed on the display unit 24. The display control unit 212 can output the aforementioned image information to the display unit 24 when a specified application installed on the user terminal 2 is started, or it can output it at other times.

[0050] The user requests the display of a recording image by operating the operation unit 25 and displaying the request to receive the image. When the display control unit 212 receives the request from the user, it sends a transmission command for the recording image to the image generating device 1 via the communication unit 23. When the display control unit 212 receives the recording image sent from the output unit 114 of the image generating device 1 as a response to the transmission command, it outputs the recording image to the display unit 24. Thus, the recording image is displayed on the display unit 24.

[0051] Figure 8 This is a flowchart illustrating an example of a process executed by the controller 10 of the image generation apparatus 1 according to a predetermined procedure. The process shown in the flowchart begins when the controller 10 is activated and is repeated at a predetermined cycle.

[0052] First, in step S1, the controller 10 acquires images of the vehicle interior and exterior via the imaging device 14. Additionally, the controller 10 acquires an image of the vehicle's exterior from the storage unit 12. In step S2, the controller 10 determines, based on the interior image, whether a specific interior condition has been detected. If step S2 is negative (S2: No), in step S3, the controller 10 determines whether a specific location has been detected within a certain time period after the specific interior condition was detected in step S2.

[0053] When step S3 is affirmative (S3: Yes), controller 10 proceeds to step S7. When step S3 is negative (S3: No), in step S4, controller 10 generates a recording image representing a specific interior condition based on the interior image.

[0054] When step S2 is negative (S2: No), in step S5, the controller 10 determines whether a specific location has been detected based on the external image. When step S5 is negative (S5: No), the controller 10 ends the process. When step S5 is positive (S5: Yes), in step S6, it is determined whether a specific interior condition has been detected within a certain period of time after the specific location was detected in step S2.

[0055] When step S6 is affirmative (S6: Yes), in step S7, the controller 10 determines whether the occupant's gaze direction is detected based on the in-vehicle interior image. When step S6 or S7 is negative (S6 or S7: No), in step S8, the controller 10 generates a recording image of the vehicle with a specific location as the background, based on the exterior and exterior images.

[0056] When step S7 is affirmative (S7: Yes), in step S9, the controller 10 generates a recording image taking into account the camera angle considering the gaze direction, based on the interior image, exterior image, and appearance image. More specifically, the controller 10 generates a recording image of the vehicle taken from the opposite side of the gaze direction at a camera angle facing the gaze direction, with a specific location as the background. In step S10, the controller 10 outputs the recording images generated in steps S7, S8, and S9 to the storage unit 12.

[0057] The following effects are obtained by adopting the implementation methods described above.

[0058] (1) The image generation apparatus 1 includes: an interior camera 14a, which acts as a first shooting device to capture images of the interior of the vehicle including the occupants; an exterior camera 14b, which acts as a second shooting device to capture images of the surroundings of the vehicle; an image acquisition unit 111, which acts as an acquisition unit to read from the storage unit 12 and acquire an exterior image of the vehicle; and an image generation unit 113, which acts as a generation unit to combine at least one of the interior image acquired by the interior camera 14a and the exterior image acquired by the exterior camera 14b with the exterior image acquired by the image acquisition unit 111 to generate a recording image. The image acquisition unit 111 reads from the storage unit 12 and acquires the exterior image. In this way, by using the recording image (composite image) to include the exterior image of the vehicle, the generated composite image can be used as an image that reproduces the actual situation or is close to the actual situation. As a result, the user can easily review the actual scene of a trip or drive based on the recording image.

[0059] (2) The image generation device 1 further includes a detection unit 112, which detects specific landmarks or specific areas existing around the vehicle based on the exterior images acquired by the exterior camera 14b. When the detection unit 112 detects a specific landmark or specific area, the image generation unit 113 generates a recording image including the vehicle and the detected specific landmark or specific area. Figure 4 , Figure 5 , Figure 6 This allows users to recall, through images, the places they actually visited during their trips or drives.

[0060] (3) The detection unit 112 also detects specific interior conditions based on the interior images. When the detection unit 112 detects specific interior conditions and specific landmarks or specific areas, the image generation unit 113 generates a recording image based on the interior images, exterior images, and exterior images. Figure 5 This allows users to review scenes from their trips and drives, including those inside the car, using images from the recorded footage.

[0061] (4) The detection unit 112 also detects the occupant's gaze direction based on the in-vehicle image. The image generation unit 113 generates a recording image of the camera angle facing the gaze direction. Figure 5 As a result, users can use images to recall their trips, landmarks they noticed while driving, and other similar experiences based on their records.

[0062] (5) The image generation unit 113 generates a recording image based on the images inside and outside the vehicle, including specific landmarks or specific areas visible through the vehicle windows from inside the vehicle and the occupants inside the vehicle. Figure 6 Thus, users can recall the actual atmosphere and feelings of a trip or drive using images from their records.

[0063] (6) The image generation unit 113 uses multimodal LLM to synthesize images of the vehicle interior, exterior, and exterior. As a result, the generation of recording images can be instructed by prompts expressed in natural language, thus enabling the generation of recording images that are more suitable for the actual scene and atmosphere.

[0064] (7) When the detection unit 112 detects a specific interior condition but the detection unit 112 does not detect a specific landmark or specific area, the image generation unit 113 generates a recording image based on the interior image instead of using the exterior image. Figure 3 Therefore, the recorded images only preserve the interior conditions at the time a specific interior situation occurred, making it easy for users to review specific interior conditions unrelated to the external environment.

[0065] The above-described embodiments can be modified in various ways. The following describes a modified example. In the above-described embodiment, the image generation unit 113 generates a recording image, including specific landmarks or specific areas visible through the vehicle windows from inside the vehicle and the occupants inside the vehicle, based on the interior and exterior images. Figure 6 However, the image generation unit can also reduce the display size of specific landmarks, etc., that are visible through the car window on the recorded image, by including them within the car window frame. Additionally, the image generation unit can also enlarge the display size of specific landmarks, etc., if the size of the specific landmarks visible through the car window on the recorded image is too small; specifically, if the proportion of the specific landmarks to the entire car window frame is less than a predetermined value.

[0066] Furthermore, in the above embodiment, the image generation unit 113 generates a recording image of the vehicle with a specific landmark or specific area as the background. Figure 5 However, when the image generation unit encounters large natural landscapes (mountains, seas, etc.) or structures (high-rise buildings, etc.) such as specific landmarks, it can also generate recording images obtained through aerial photography by drones, etc., by including the natural landscape or structure entirely within the image. Furthermore, when aiming to include a specific landmark or area entirely within the image, if the visibility of the vehicle deteriorates—specifically, if the vehicle's proportion in the overall image is less than a predetermined value—the display size of the vehicle can be enlarged. Alternatively, maintaining the vehicle's proportion in the overall image at a predetermined value or higher can be prioritized over including the specific landmark or area entirely within the image.

[0067] From another perspective, the image generation apparatus described above can also be configured as an image generation method, which includes the following steps: acquiring an interior image of a vehicle from a first imaging device that captures images of the interior of the vehicle including the occupants; acquiring an exterior image of the vehicle from a second imaging device that captures images of the area around the vehicle; acquiring an exterior image of the vehicle stored in a storage unit; and combining at least one of the interior image and the exterior image with the exterior image to generate a recording image.

[0068] The present invention can also be constructed by replacing the above-described image generation method with a program that causes a computer to execute a process to generate a recording image for use by passengers to recall memories of traveling in the vehicle, such as for a drive. Furthermore, the present invention can also be constructed by replacing it with a computer-readable storage medium on which the program is recorded.

[0069] The above description is merely an example, and the present invention is not limited to the above embodiments and modifications as long as it does not impair the characteristics of the invention. One or more of the above embodiments and modifications can be combined in any way, and modifications can also be combined with each other.

[0070] Using this invention, the generated synthetic image can be used as an image that reproduces or closely approximates the actual situation, and can be used to generate recording images suitable for review and recall.

[0071] The present invention has been described above in conjunction with preferred embodiments, but those skilled in the art should understand that various modifications and changes can be made without departing from the scope of the claims.

Claims

1. An image generation apparatus, characterized in that, have: The first camera device captures images of the passenger compartment; The second camera device photographs the area around the vehicle. The acquisition unit (111) acquires an image of the exterior of the vehicle; and The generation unit (113) combines at least one of the interior image of the vehicle obtained by the first shooting device and the exterior image of the vehicle obtained by the second shooting device with the exterior image obtained by the acquisition unit (111) to generate an image for recording.

2. The image generation apparatus according to claim 1, characterized in that, It also includes a storage unit (12) for storing the appearance image. The acquisition unit (111) reads and acquires the appearance image from the storage unit (12).

3. The image generation apparatus according to claim 1, characterized in that, It also includes a detection unit (112) that detects specific landmarks or specific areas existing around the vehicle based on the exterior images acquired by the second imaging device. When the detection unit (112) detects the specific landmark or the specific area, the generation unit (113) generates the recording image including the vehicle and the detected specific landmark or the specific area.

4. The image generation apparatus according to claim 3, characterized in that, The detection unit (112) also detects specific interior conditions based on the interior images. When the detection unit (112) detects the specific interior condition of the vehicle and the specific landmark or the specific area, the generation unit (113) generates the recording image based on the interior image, the exterior image, and the exterior image.

5. The image generation apparatus according to claim 4, characterized in that, The detection unit (112) also detects the occupant's gaze direction based on the in-vehicle image. The generation unit (113) generates the recording image at the camera angle facing the gaze direction.

6. The image generation apparatus according to claim 1, characterized in that, It also includes a detection unit (112) that detects specific landmarks or specific areas existing around the vehicle based on the exterior images acquired by the second imaging device. The generation unit (113) generates the recording image based on the interior image and the exterior image, including the specific landmark or specific area visible through the window from inside the vehicle and the occupants inside the vehicle.

7. The image generation apparatus according to claim 1, characterized in that, The generation unit (113) uses a multimodal LLM (large language model) to synthesize the interior image, the exterior image, and the appearance image.

8. The image generation apparatus according to claim 3, characterized in that, The image used for recording is the first image used for recording. When the specific landmark and the specific area are not detected by the detection unit (112), the generation unit (113) does not use the exterior image, but generates a second recording image based on the interior image when the specific interior condition is detected by the detection unit (112).

9. An image generation method, characterized in that, The steps include the following: The first imaging device acquires an image of the vehicle interior, which is captured by the first imaging device to capture images of the vehicle interior including the occupants. The second shooting device acquires images of the exterior of the vehicle obtained by the second shooting device that captures images of the area around the vehicle. Obtain an exterior image of the vehicle stored in the storage unit; as well as At least one of the interior image and the exterior image is combined with the exterior image to generate a recording image.

10. A storage medium capable of being read by a computer and used for storing an image generation program, characterized in that, The image generation program causes the computer to perform the following steps: The first imaging device acquires an image of the vehicle interior, which is captured by the first imaging device that captures images of the vehicle interior containing the occupants. The second shooting device acquires images of the exterior of the vehicle obtained by the second shooting device that captures images of the area around the vehicle. Obtain an exterior image of the vehicle stored in the storage unit; as well as At least one of the interior image and the exterior image is combined with the exterior image to generate a recording image.

Images