Image generating apparatus

Abstract

To improve a technique for generating a composite image.SOLUTION: The image generation device 20 includes a control unit that acquires a vehicle exterior image obtained by imaging the periphery of the vehicle VH and a vehicle interior image obtained by imaging the interior of the vehicle VH, analyzes the acquired vehicle interior image, and generates a composite image obtained by combining the vehicle exterior image and the vehicle interior image based on the analysis result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to image generation devices. [Background technology]

[0002] Patent Document 1 discloses a technique for generating a composite image by combining an image of a vehicle and an image of the exterior of the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-040497 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology of Patent Document 1 does not attempt to adjust the timing of combining an exterior image of the vehicle's surroundings with an interior image of the vehicle's interior based on the interior image, so there is room for improvement in the technology for generating composite images.

[0005] An object of the present disclosure is to improve techniques for generating synthetic images.

[0006] An image generating device according to an embodiment of the present disclosure includes: Acquire an exterior image of the vehicle's surroundings and an interior image of the vehicle's interior, The vehicle control unit analyzes the acquired vehicle interior image and generates a composite image by combining the vehicle exterior image and the vehicle interior image based on the analysis result. [Effects of the Invention]

[0007] In accordance with the present disclosure, techniques for generating composite images are improved. [Brief explanation of the drawings]

[0008] [Figure 1]1 is a block diagram illustrating a configuration of a system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating a configuration of an image generating device according to an embodiment of the present disclosure. [Figure 3] 10 is a flowchart illustrating an operation of the image generating device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0010] In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0011] The configuration of a system 10 according to this embodiment will be described with reference to FIG.

[0012] The system 10 includes an image generating device 20 mounted on a vehicle VH. The system 10 may also include an external device 30.

[0013] The image generation device 20 is capable of communicating with an external device 30 via a network 40 .

[0014] The image generation device 20 includes a microcomputer equipped with a processor, a memory, an input / output interface, etc. The microcomputer is also called an ECU and controls various functions of the vehicle VH. "ECU" is an abbreviation for Electronic Control Unit.

[0015] The external device 30 is a computer such as a server that is installed in a facility such as a data center and that belongs to a cloud computing system or other computing system.

[0016] The vehicle VH may be any type of automobile, such as a gasoline vehicle, a diesel vehicle, a hydrogen vehicle, an HEV, a PHEV, a BEV, or an FCEV. "HEV" is an abbreviation for hybrid electric vehicle. "PHEV" is an abbreviation for plug-in hybrid electric vehicle. "BEV" is an abbreviation for battery electric vehicle. "FCEV" is an abbreviation for fuel cell electric vehicle. The vehicle VH has one or more passengers P on board. The passengers P include passengers P1 and P2. In this embodiment, the vehicle VH is a private car, but is not limited to this and may also be a taxi, a bus, or the like. The vehicle VH may be driven by a driver, or may be an AV with an arbitrary level of automated driving. "AV" is an abbreviation for autonomous vehicle.

[0017] Network 40 may include the Internet, at least one WAN, at least one MAN, or any combination thereof. "WAN" is an abbreviation for wide area network. "MAN" is an abbreviation for metropolitan area network.

[0018] An overview of this embodiment will be described with reference to FIG.

[0019] Taking commemorative photos while driving is a hassle because you have to stop the vehicle and get out before taking the photo. Also, stopping on the shoulder of a highway is difficult for safety reasons.

[0020] In this embodiment, the image generating device 20 acquires an exterior image M1 of the vehicle VH and an interior image M2 of the vehicle VH. The image generating device 20 analyzes the acquired interior image M2 and generates a composite image M3 by combining the exterior image M1 and the interior image M2 based on the analysis result.

[0021] According to this embodiment, the image generating device 20 generates a composite image M3 based on the results of analyzing the vehicle interior image M2. Therefore, the timing of generating the composite image M3 can be adjusted based on the results of analyzing the vehicle interior image M2. As an example, by analyzing the vehicle interior image M2, the facial expression of the passenger P captured in the vehicle interior image M2 can be determined, and the composite image M3 can be generated at the timing when the facial expression is determined to be a "smile." Therefore, the composite image M3 can be generated at a timing appropriate for the passenger P of the vehicle VH. Furthermore, the passenger P can take a commemorative photo without stopping the vehicle VH and getting out. As a result, the technology for generating composite images is improved.

[0022] The configuration of an image generating device 20 according to this embodiment will be described with reference to FIG.

[0023] The image generating device 20 includes a control unit 21, a storage unit 22, and a communication unit 23. The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof, and executes predetermined processing.

[0024] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The storage unit 22 stores data used in the operation of the image generation device 20 and data obtained by the operation of the image generation device 20. In this embodiment, the storage unit 22 stores a host vehicle image M4, which is an image of the vehicle VH. The host vehicle image M4 is, for example, an image captured from a bird's-eye view from the front to the rear of the vehicle VH. The host vehicle image M4 may be a CG image. "CG" is an abbreviation for computer graphics.

[0025] The communication unit 23 includes at least one communication interface. The communication unit 23 receives data used in the operation of the image generation device 20 and transmits data obtained by the operation of the image generation device 20. In this embodiment, the communication unit 23 communicates with the external device 30.

[0026] The functions of the image generation device 20 are realized by executing a program according to this embodiment on a processor serving as the control unit 21. That is, the functions of the image generation device 20 are realized by software. The program causes a computer to execute the operations of the image generation device 20, thereby causing the computer to function as the image generation device 20. That is, the computer functions as the image generation device 20 by executing the operations of the image generation device 20 in accordance with the program.

[0027] The operation of the image generating device 20 according to this embodiment will be described with reference to Fig. 3. This operation corresponds to the image generating method according to this embodiment. That is, the image generating method according to this embodiment includes steps S1 to S5 shown in Fig. 3. Hereinafter, each step in the flowchart will be identified by an S and a number.

[0028] In step S1, the control unit 21 of the image generating device 20 acquires an outside-vehicle image M1. The outside-vehicle image M1 is an image of the periphery of the vehicle VH. In this embodiment, the term "image" also includes video. The outside-vehicle image M1 may be, for example, a three-dimensional projection of the environment surrounding the vehicle VH, or a single-image image of the environment captured by a single camera. The outside-vehicle image M1 may be acquired using any procedure, such as the following procedure. The vehicle VH is equipped with an outside-vehicle camera that captures images of the front, left, right, or rear of the vehicle VH. The outside-vehicle camera may capture images at any timing, but may be triggered, for example, by a switch operation or voice command by the passenger P. Alternatively, the control unit 21 may acquire position information of the vehicle VH based on a positioning signal from a GPS installed in the vehicle VH, and control the outside-vehicle camera to capture images when the position indicated by the position information matches a predetermined point on a map. "GPS" is an abbreviation for Global Positioning System. The control unit 21 acquires an outside-of-vehicle image M1 captured by an outside-vehicle camera. The outside-of-vehicle image M1 may include data indicating the date, time, and / or location at which the image was captured.

[0029] In S2, the control unit 21 of the image generating device 20 acquires an interior image M2. The interior image M2 is an image captured of the interior of the vehicle VH. The interior image M2 may be acquired by any procedure, for example, by the following procedure. The interior camera is a camera that captures images of the interior of the vehicle VH, and is, for example, a camera installed to capture images of the interior of the vehicle, such as an electronic mirror such as an electronic rearview mirror, or a drive recorder. Alternatively, the interior camera may be a terminal device such as a smartphone held by the passenger P, as long as it is capable of capturing images of the interior of the vehicle VH. The control unit 21 acquires the interior image M2 captured by the interior camera. In this embodiment, the interior image M2 captures the passengers P1 and P2.

[0030] As a modified example of this embodiment, the control unit 21 of the image generating device 20 may acquire, as the interior image M2, an image of the occupant P cut out from an image captured of the interior of the vehicle VH. Furthermore, the control unit 21 may acquire, as the interior image M2, an image in which a decorative icon such as a ribbon or sunglasses is superimposed on the cut-out image of the occupant P.

[0031] In S3, the control unit 21 of the image generating device 20 analyzes the vehicle interior image M2 acquired in S2. Specifically, the control unit 21 analyzes the vehicle interior image M2 to identify at least one of the facial orientation, facial expression, and movement of one or more passengers P appearing in the vehicle interior image M2.

[0032] In S4, the control unit 21 of the image generating device 20 determines whether the analysis result in S3 satisfies a preset condition. Specifically, the control unit 21 determines any one of the facial orientation, facial expression, and movement of one or more passengers P identified in S3 and appearing in the interior image M2. For example, assume that the condition is set to "all passengers facing forward." In this case, the control unit 21 determines the facial orientation identified for passengers P1 and P2 appearing in the interior image M2, and determines that the condition is met if both passengers P1 and P2 face forward. Alternatively, assume that the condition is set to "smile." In this case, the control unit 21 determines the facial expression identified for each of passengers P1 and P2 and determines whether they are smiling. The determination of whether they are smiling can be performed by determining the "smile level" of the recognized face based on image characteristics such as narrow eyes and upturned corners of the mouth. As an additional condition, a threshold value may be set for the "smile level," and the control unit 21 may determine that the condition is met when the smile level of either the passenger P1 or P2, or the average of the smile levels of the passengers P, exceeds the threshold value. Alternatively, the condition may be set as "making a peace sign." In this case, the control unit 21 determines the identified gesture of the passengers P1 and P2 captured in the in-vehicle image M2, and determines that the condition is met when the gesture is one in which the palm is facing out and the index finger and middle finger are extended in a V-shape. If it is determined that the condition is met in S4, the process proceeds to S5. On the other hand, if it is determined that the condition is not met in S4, the process returns to S1, and the processes from S1 to S4 are repeated until the control unit 21 determines that the condition is met. Alternatively, the process may return to S2 instead of S1, and only the in-vehicle image M2 is acquired again, and the process up to S4 may be similarly repeated.

[0033] In S5, the control unit 21 of the image generating device 20 generates a composite image M3 by combining the vehicle exterior image M1 and the vehicle interior image M2. Specifically, the control unit 21 generates the composite image M3 by superimposing the vehicle interior image M2 on the vehicle exterior image M1. The composite image M3 may be a still image or a moving image. The control unit 21 controls the display device, such as a display, provided in the cabin of the vehicle VH to display the generated composite image M3. At this time, the control unit 21 may receive an instruction from at least one of the passengers P and control the adjustment of the imaging range of the vehicle exterior image M1 and / or the vehicle interior image M2 based on the received instruction. Specifically, the control unit 21 receives any operation, such as a touch operation, a gesture operation, or an operation using a controller, performed by the passenger P on the display device as an instruction from the passenger P. Alternatively, the control unit 21 may receive a voice uttered by the passenger P as an instruction through voice recognition. The control unit 21 adjusts the imaging range based on an instruction from the passenger P, for example, by changing the virtual viewpoint of the vehicle exterior image M1 and / or the arrangement, color tone, and / or size of the vehicle interior image M2. The passenger P may select any viewpoint and composite pattern for the vehicle exterior image M1 as the background. This configuration makes it possible to generate a composite image that suits the preferences of the passenger P as a user.

[0034] As described above, the control unit 21 of the image generating device 20 acquires an exterior image M1 capturing an image of the periphery of the vehicle VH and an interior image M2 capturing an image of the interior of the vehicle VH. The control unit 21 analyzes the acquired interior image M2 and generates a composite image M3 by combining the exterior image M1 and the interior image M2 based on the analysis result.

[0035] According to this embodiment, a composite image can be generated using images acquired at a timing appropriate for a driver and passenger P to take a photo, thereby improving the technique for generating a composite image.

[0036] The control unit 21 of the image generating device 20 may further acquire a host vehicle image M4 and further composite the acquired host vehicle image M4 onto the composite image M3. Specifically, the control unit 21 reads the host vehicle image M4 from the storage unit 22 and superimposes the host vehicle image M4 on the composite image M3. The storage unit 22 may store, as the host vehicle image M4, a plurality of images of the vehicle VH captured at different times of the day or under different weather conditions, and the control unit 21 may select, from the plurality of host vehicle images M4, an image that matches the time of day or weather conditions during which the vehicle exterior image M1 obtained in S1 was captured. This configuration makes it possible to generate a composite image that further suits the user's preferences.

[0037] The control unit 21 of the image generating device 20 may further transmit the composite image M3 generated in S5 to the external device 30 via the communication unit 23. The control unit 21 may acquire data indicating the date, time, and / or location at which the vehicle exterior image M1 or the vehicle interior image M2 was captured and transmit the acquired data as photo spot information D1 to the external device 30 together with the composite image M3. Alternatively, the control unit 21 may upload the composite image M3 together with the photo spot information D1 to the cloud. With this configuration, the passenger P can acquire the composite image M3 and the photo spot information D1 by accessing the external device 30 from an external device such as his / her own smartphone or by downloading them from the cloud via a two-dimensional barcode such as a QR code (registered trademark). Furthermore, third parties other than the passenger P can also utilize the composite image M3 and the location and date information at which the image was captured. Furthermore, the passenger P can share the composite image M3 and the photo spot information D1 with third parties.

[0038] In this embodiment, the image generating device 20 is described as being mounted on the vehicle VH, but the image generating device 20 may be installed outside the vehicle VH, and the image generating device 20 and the vehicle VH may generate the composite image M3 while transmitting and receiving various data via the network 40.

[0039] Furthermore, in this embodiment, the photographing by the outside camera in S1 has been described as being triggered by the passenger P operating a switch, operating by voice, or matching positions, etc., but the control unit 21 may first perform the processes of S2 to S4 prior to S1, and then perform the processes of S1 and S5 when at least one of the passenger P's facial direction, expression, and movement satisfies a preset condition.

[0040] The present disclosure is not limited to the above-described embodiments. For example, multiple blocks shown in the block diagrams may be integrated, or a single block may be divided. Instead of executing multiple steps shown in the flowcharts in chronological order as described, steps may be executed in parallel or in a different order depending on the processing capabilities of the device executing each step, or as needed. Other modifications are possible within the scope of the present disclosure. [Explanation of symbols]

[0041] 10 Systems 20 Image generation device 21 Control section 22 Memory section 23 Communications Department 30 External Devices 40 Network VH vehicle P, P1, P2 passengers

Claims

1. Acquire an exterior image of the vehicle and an interior image of the vehicle; An image generating device including a control unit that analyzes the acquired vehicle interior image and generates a composite image by combining the vehicle exterior image and the vehicle interior image based on the analysis result.

2. The image generating device according to claim 1, wherein the control unit generates the composite image when, as a result of analyzing the in-vehicle image, at least one of the facial orientation, facial expression, and movement of one or more passengers appearing in the in-vehicle image satisfies a predetermined condition.

3. The image generating device according to claim 2 , wherein the control unit receives an instruction from at least one of the passengers and performs control to adjust the imaging range of the vehicle outside image and / or the vehicle inside image based on the received instruction.

4. a communication unit that communicates with an external device outside the image generating apparatus, The image generating device according to claim 1 , wherein the control unit transmits the composite image to the external device via the communication unit together with data indicating the date, time, and / or location at which the vehicle exterior image or the vehicle interior image was captured.

5. The image generating device according to claim 1 , wherein the control unit further acquires a host vehicle image that is an image of the vehicle, and further composites the acquired host vehicle image onto the composite image.

Images