Vehicle service provision system

The system processes occupant images to generate a morphed face image with high similarity, addressing legal restrictions and ensuring accurate passenger condition assessment in vehicle service systems.

JP7594693B2Active Publication Date: 2024-12-04SUBARU CORP

Patent Information

Application Number
JP2023579030
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-09-11
Publication Date
2024-12-04
Estimated Expiration
2042-09-11

AI Technical Summary

Technical Problem

Existing vehicle service providing systems face challenges in transmitting occupant images to server devices due to legal restrictions on personally identifiable information, particularly in Europe, while also needing to accurately grasp the condition and facial expressions of passengers for services like emergency response and monitoring.

Method used

A vehicle service providing system that uses an in-vehicle camera to capture occupant images, processes them to generate a morphed face image by selecting a sample face image with high similarity in brightness values, and transmits this instead of the original image to protect personal information.

Benefits of technology

The system accurately reflects passenger facial expressions and conditions without transmitting actual faces, ensuring compliance with legal restrictions and maintaining service effectiveness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007594693000001
    Figure 0007594693000001
  • Figure 0007594693000002
    Figure 0007594693000002
  • Figure 0007594693000003
    Figure 0007594693000003
Patent Text Reader

Abstract

[Problem] To protect personal information such that the convenience of a vehicle service provision system does not suffer extreme impairment. [Solution] A vehicle control unit 65 of this vehicle service provision system 1 generates luminance for the face of an occupant in a captured image acquired by an in-vehicle camera 55. The vehicle control unit 65 assesses and selects, from a plurality of acquired sample facial images, an image having the highest degree of similarity of luminance. The vehicle control unit 65 performs a morphing process for making the selected sample facial image more similar to the face of the occupant, and generates a morphing facial image and transmits the same. A server control unit 18 of a server device 5 of the vehicle service provision system 1 outputs the morphing facial image generated by the morphing process from an output unit 13 in place of the image of the face of the occupant.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a vehicle service providing system. [Background technology]

[0002] In a vehicle such as an automobile, malfunctions may occur in the vehicle itself or in the passengers while the vehicle is in motion. Additionally, it may be desirable for vehicles and their occupants to receive various content services while traveling. These vehicle services may include emergency response services to deal with vehicle or occupant malfunctions, occupant monitoring services for that purpose, and content provision services. In such a vehicle service providing system, in order to provide a vehicle service, information is basically transmitted from the vehicle to a server device, and the server device provides the service based on the information received. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-043571 [Patent Document 2] JP 2014-067131 A Summary of the Invention [Problem to be solved by the invention]

[0004] In such a vehicle service providing system, particularly in the above-mentioned emergency response service and occupant monitoring service, it is desirable to transmit images captured by an in-vehicle camera showing the status of the occupants from the vehicle to a server device. By displaying and outputting the images captured by the in-vehicle camera on the server device, the operator can accurately grasp the current status and facial expression of the occupants from the captured images of the occupants, and as a result, it becomes possible to provide a service optimal for the current status of the occupants.

[0005] On the other hand, when transmitting information from a vehicle to a server device in this manner, depending on the information, it may be restricted based on laws and regulations, etc. For example, in Europe, there are strict restrictions imposed independently on the transmission and reception of information that can identify an individual. Furthermore, companies that wish to protect such personal information will wish to independently impose appropriate restrictions on the information transmitted from vehicles to server devices. The captured image of the vehicle occupant includes image components of the occupant's face, which is personal information and should be appropriately protected.

[0006] Patent Documents 1 and 2 disclose abstracting or replacing image components of the occupant's face included in a captured image. However, if the image components of the occupant's face are abstracted or replaced as in Patent Documents 1 and 2, even if the image is output to a server device, it is difficult for an operator to accurately grasp the occupant's current condition, facial expression, etc. from the image. In particular, in emergency response services and passenger monitoring services, it is desirable to be able to check the state of injuries on the passenger's face, etc. This cannot be done with abstracted or replaced images.

[0007] In this manner, it is desirable for the vehicle service providing system to protect personal information while minimizing the loss of convenience. [Means for solving the problem]

[0008] A vehicle service providing system according to one embodiment of the present invention is a system that transmits information from a vehicle to a server device and provides a service based on the information received by the server device, wherein the vehicle has an in-vehicle camera capable of capturing an image of an occupant of the vehicle and a vehicle control unit, the server device has an image output unit and a server control unit, and the vehicle control unit performs an occupant image acquisition process for acquiring an image including the face of the occupant from the in-vehicle camera, a luminance value generation process for generating a luminance value of the occupant's face in the acquired image, a sample image acquisition process for acquiring a plurality of sample face images of a plurality of people different from the occupant to be cut out from the captured image, a determination process for determining a similarity between the luminance values ​​of the acquired plurality of sample face images and the luminance value of the occupant's face generated by the luminance value generation process, and the determination process a selection process for selecting a sample face image having the highest similarity in brightness value, and a morphing process for performing a morphing process on the selected sample face image to bring it closer to the face of the occupant in the acquired captured image, thereby generating a morphed face image, and transmitting the morphed face image generated by the process to the server device in place of the captured image including the face of the occupant, and the server control unit uses information received from the vehicle control unit of the vehicle to perform processes among the brightness value generation process, the sample image acquisition process, the judgment process, the selection process, and the morphing process that are not performed by the vehicle control unit, and outputs the morphed face image generated by the morphing process from the output unit in place of the image of the face of the vehicle occupant. Effect of the Invention

[0009] In the vehicle service providing system of the present invention, the vehicle control unit of the vehicle and the server control unit of the server device execute the occupant image acquisition process, the brightness value generation process, the sample image acquisition process, the determination process, the selection process, and the morphing process. Then, the server control unit outputs the morphed face image obtained by the morphing process from the output unit instead of the facial image of the vehicle occupant. This allows the output unit to output a facial image reflecting the state of the occupant, such as the facial expression. As a result, the operator can grasp the current actual state and facial expression of the occupant based on the output facial image. It is difficult for the operator to accurately grasp the current actual state and facial expression of the occupant, as in the case where the image components of the occupant's face are abstracted, replaced, or masked. In particular, in the present invention, instead of performing a morphing process to bring a single fixed face image different from the occupant closer to the occupant's face, a sample face image that has a high similarity to the occupant's face brightness value is selected from among a plurality of sample face images. As a result, the face image of the vehicle occupant output from the output unit of the server device can be a face image that is close to the occupant's face brightness value depending on the occupant. As a result, unlike the case where a single fixed face image of another person is always output, it is considered that the operator can grasp the mood of the occupant and more accurately judge the state of the occupant based on the output of a face image corresponding to the occupant. In addition, the vehicle control unit of the vehicle executes the passenger image acquisition process, the cut-out process, the point cloud generation process, the patch generation process, the sample image acquisition process, and the morphing process, and transmits the morphed face image generated by the process to the server device instead of the captured image including the passenger's face. In this way, in the present invention, the passenger's face image itself, which needs to be protected as the passenger's personal information, can be prevented from being transmitted to the outside. In this way, in the present invention, by selecting and morphing a sample face image that is similar to the brightness value of the actual occupant's face, it is possible to protect personal information while minimizing the loss of convenience of the vehicle service provision system. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a system configuration diagram of a vehicle service providing system according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is an explanatory diagram of a basic communication procedure between the automobile of FIG. 1 and a server device in an example in which an emergency response service is provided in the vehicle service providing system of FIG. [Diagram 3] FIG. 3 is a configuration diagram of the server device of FIG. [Figure 4] FIG. 4 is a block diagram of the control system of the automobile shown in FIG. [Diagram 5] FIG. 5 is a configuration diagram of various control devices shown in FIG. [Figure 6] FIG. 6 is a flowchart of the control of transmission of information for emergency response services by the control system of the automobile of FIG. [Figure 7] FIG. 7 is a detailed flowchart of the processed image generation control in step ST15 in FIG. [Figure 8] FIG. 8 is an explanatory diagram of a facial image processing procedure in the first embodiment. [Figure 9] FIG. 9 is an explanatory diagram of an example of an image captured by the in-vehicle camera of FIG. [Figure 10] FIG. 10 is an explanatory diagram of an example of a face image that is an imaging region of the driver's face cut out from the captured image of FIG. [Figure 11] FIG. 11 is a diagram illustrating an example of point cloud information generated for the face image of the driver in FIG. [Figure 12] FIG. 12 is an explanatory diagram of an example of patch information generated for the face image of the driver in FIG. [Figure 13] FIG. 13 is an explanatory diagram of an example of a combination of a plurality of sample face images. [Figure 14] FIG. 14 is an explanatory diagram of an example of point cloud information generated for the sample face image selected in FIG. [Figure 15] FIG. 15 is an explanatory diagram of an example of patch information generated for the sample face image selected in FIG. [Figure 16]FIG. 16 is an explanatory diagram of an example of a morphed face image that is transmitted from the automobile to the outside as a face image of an occupant. [Figure 17] FIG. 17 is a detailed flowchart of the processed image generation control in the second embodiment of the present invention. [Figure 18] FIG. 18 is an explanatory diagram of an example of a face image which is an imaging area of ​​the driver's face in this embodiment. [Figure 19] FIG. 19 is a diagram illustrating an example of point cloud information and patch information generated for the face image of the driver in FIG. [Figure 20] FIG. 20 is an explanatory diagram of an example of point cloud information and patch information generated for a sample face image selected from among a plurality of sample face images. [Figure 21] FIG. 21 is an explanatory diagram of an example of a morphed face image that is transmitted from the automobile to the outside as a face image of an occupant. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] [First embodiment] FIG. 1 is a configuration diagram of a vehicle service providing system 1 according to a first embodiment of the present invention. 1 includes an automobile 2 and a server device 5 for providing vehicle services to the automobile 2. The automobile 2 and the server device 5 transmit and receive information to and from each other via a communication network 6. Here, the automobile 2 is an example of a vehicle, and may basically be one in which multiple passengers can ride. In an automobile 2 in which multiple passengers can ride, the driver and the assistant driver can sit side by side in the vehicle width direction. However, the vehicle to which the present invention is applicable is not limited to an automobile 2 having such characteristics.

[0013] 1 includes a plurality of base stations 3 arranged along the travel route of the automobile 2, and a communication network 4 to which a server device 5 is connected together with the plurality of base stations 3. The base stations 3 may be, for example, base stations of a carrier communication network for mobile terminals or the like, or base stations for ITS services or ADAS services to the automobile 2. Some fifth-generation base stations of the carrier communication network have high-performance information processing functions. In this case, the server device 5 can be provided in a distributed manner across the plurality of base stations 3.

[0014] While the automobile 2 is traveling, a malfunction may occur in the automobile 2 itself or in a passenger. Furthermore, passengers may use various content services while traveling in the automobile 2. Content services include, for example, music distribution services, video distribution services, and the like. The vehicle service providing system 1 can provide these vehicle services to the automobile 2 by using a server device 5 that transmits and receives information to and from the automobile 2. The vehicle service providing system 1 may provide, for example, an emergency response service for dealing with malfunctions of the automobile 2 or its occupants, and a monitoring service therefor. In addition, the vehicle service providing system 1 may provide content information from the server device 5.

[0015] FIG. 2 is an explanatory diagram of a basic communication procedure between the automobile 2 and the server device 5 in FIG. 1 in an example in which an emergency response service is provided in the vehicle service providing system 1 in FIG. 2 shows an automobile 2 and a server device 5. Time flows from top to bottom.

[0016] 2, the automobile 2 acquires information on the automobile 2 and the occupants in step ST1, and transmits the acquired information to the server device 5 in step ST2. The information transmitted from the automobile 2 to the server device 5 may include information indicating the state of the automobile 2, information indicating the state of the occupants, etc. The server device 5 waits to receive information from the automobile 2, as shown in step ST3. Then, when the server device 5 receives information from the automobile 2, it outputs the received information by displaying it in step ST4. This allows the operator to check the information of the automobile 2 and its occupants. Then, when it is determined that confirmation with the occupants is necessary, the operator operates the server device 5. After outputting the received information, the server device 5 determines whether communication is necessary in step ST5. When the operator operates the server device 5, the server device 5 determines that communication is necessary, and starts communication with the automobile 2 in step ST6. This establishes a communication path between the automobile 2 and the server device 5, as shown in step ST7, allowing the operator and the occupants to talk by voice or video. Then, when it is determined from the call with the occupant that an emergency response is necessary, the operator operates the server device 5. The server device 5 makes an emergency call to the road service 9 as shown in step ST8. Based on the call, the road service 9 dispatches to the automobile 2 as shown in step ST9. The road service 9 provides a vehicle service for emergency response. This allows the automobile 2 or the occupants to enjoy the emergency response services provided by the vehicle service providing system 1.

[0017] In this manner, in the vehicle service providing system 1, in order to provide a vehicle service, information is basically transmitted from the automobile 2 to the server device 5, and the server device 5 provides the service based on the information received. Incidentally, in such a vehicle service providing system 1, particularly in the above-mentioned emergency response service and occupant monitoring service, it is desirable to transmit images captured by the in-vehicle camera 55 showing the status of the occupants from the automobile 2 to the server device 5. By displaying and outputting the images captured by the in-vehicle camera 55 on the server device 5, the operator can accurately grasp the current status and facial expression of the occupants from the captured images of the occupants. As a result, the operator can provide the most suitable service according to the current status of the occupants.

[0018] On the other hand, when transmitting information from the automobile 2 to the server device 5 in this manner, depending on the information, the information may be restricted based on laws and regulations, etc. For example, in Europe, there are strict restrictions imposed independently on the transmission and reception of information that can identify an individual. Furthermore, companies that wish to protect such personal information may consider voluntarily restricting the information transmitted from automobile 2 to server device 5. The captured images of the occupants of the automobile 2 described above include image components of the faces of the occupants. It is desirable that the facial photographs of the occupants, like the names of the occupants, be protected appropriately as personal information in accordance with legal requirements and other requirements.

[0019] In this way, it is desirable for the vehicle service providing system 1 to protect personal information while minimizing the loss of convenience.

[0020] FIG. 3 is a configuration diagram of the server device 5 of FIG. 3 includes a server timer 16, a server memory 17, a server CPU 18, and a server bus 19 to which these are connected. In addition, the server bus 19 may also be connected to, for example, a server GNSS receiver 11, a server communication device 12, a server display device 13, a server operation device 14, a server audio device 15, and the like.

[0021] The server communication device 12 is connected to a communication network 4 of a communication network 6. The server communication device 12 transmits and receives information to and from other devices connected to the communication network 4, such as an automobile 2 and a base station 3, via the communication network 4. The server GNSS receiver 11 receives radio waves from a GNSS satellite 110 shown in FIG. 1 to obtain the current time. The server display device 13 displays and outputs the information of the server device 5. The server display device 13 may be, for example, a liquid crystal monitor. The server display device 13 can function as an image output unit in the server device 5. The server operation device 14 is a device operated by an operator in the server device 5. The server operation device 14 may be, for example, a keyboard, a touch panel, or the like. The server audio device 15 is a device used by an operator for calls in the server device 5. The server audio device 15 may be configured with, for example, a speaker and a microphone. The server timer 16 measures time or hour. The time of the server timer 16 may be calibrated by the current time acquired by the server GNSS receiver 11. The server memory 17 records the programs executed by the server CPU 18 and various information. The server memory 17 may be composed of, for example, a semiconductor memory, a HDD, or the like. The semiconductor memory includes, for example, a volatile memory such as a RAM, and a non-volatile memory such as a ROM or SSD. A volatile memory temporarily records information, and is suitable for recording, for example, personal information.

[0022] The server CPU 18 reads and executes the programs recorded in the server memory 17. In this way, the server CPU 18 functions as a server control unit that controls the overall operation of the server device 5. The server CPU 18 as a server control unit may, for example, manage and control temporary recording of personal information in the server memory 17. For example, the server CPU 18 may delete from the server memory 17 personal information for which the provision of the vehicle service has ended. Furthermore, the server CPU 18 may manage and control the transmission and reception of information to and from the automobile 2 using the server communication device 12. For example, when the server communication device 12 receives information from the automobile 2, the server CPU 18 may execute control according to the received information, for example, control for vehicle services. This allows the server device 5 to receive information from the automobile 2 and provide vehicle services based on the received information.

[0023] FIG. 4 is a configuration diagram of the control system 20 of the automobile 2 of FIG. The control system 20 of the automobile 2 in Fig. 4 has a vehicle network 30 and a plurality of control devices connected to the vehicle network 30. Fig. 4 shows, as examples of the plurality of control devices, a drive control device 21, a steering control device 22, a braking control device 23, a driving control device 24, a driving operation control device 25, a detection control device 26, a communication control device 27, and a telephone call control device 28. In addition to the above, the automobile 2 may also be provided with an air conditioning control device, an occupant monitoring control device, a driving position control device, and the like.

[0024] The vehicle network 30 may be compliant with standards such as CAN (Controller Area Network) or LIN (Local Interconnect Network) that are used in the automobile 2. Such a vehicle network 30 has a plurality of bus cables 32 to which a plurality of control devices are connected, and a central gateway device 31 as a relay device to which the plurality of bus cables 32 are connected. The central gateway device 31 controls routing between the plurality of control devices via the plurality of bus cables 32. This allows the control devices to transmit and receive information to and from other control devices connected to other bus cables 32.

[0025] The driving operation control device 25 is connected to operation members operated by the driver to drive the automobile 2, such as a steering wheel 41, a brake pedal 42, an accelerator pedal 43, and a shift lever 44. The driving operation control device 25 detects the driver's operation of the operation members, generates operation information according to the operation, and outputs it to other control devices via the vehicle network 30.

[0026] The driving control device 24 controls the driving of the automobile 2. The driving control device 24 acquires operation information from the driving operation control device 25 through the vehicle network 30, for example. Then, the driving control device 24 generates driving information, steering information, and braking information according to the acquired information, and outputs them to the driving control device 21, the steering control device 22, and the braking control device 23 through the vehicle network 30. At this time, the driving control device 24 may generate driving information, steering information, and braking information that directly correspond to the operation information, or may generate driving information, steering information, and braking information that are adjusted to assist the operation. In addition, the driving control device 24 may generate driving information, steering information, and braking information for automatic driving that is not operated by the occupant.

[0027] The drive control device 21 acquires drive information from the vehicle network 30 and controls the magnitude and balance of the drive force of the automobile 2 by controlling the engine, motor, transmission, etc. (not shown), which are the power sources of the automobile 2 . The steering control device 22 acquires steering information from the vehicle network 30, and controls a steering motor (not shown) of the automobile 2, etc., to control the traveling direction. The braking control device 23 acquires braking information from the vehicle network 30 and controls braking members and a motor (not shown) of the automobile 2 to control the braking of the automobile 2 . The communication control device 27 communicates with the base station 3 located near the automobile 2 to establish a wireless communication path. The communication control device 27 transmits and receives information to and from the server device 5 and the like using the wireless communication path established with the base station 3. A speaker 46 and a microphone 47 are connected to the call control device 28, which are used by the passengers of the automobile 2 to communicate with, for example, an operator of the server device 5.

[0028] The detection control device 26 is connected to various sensor members provided in the automobile 2. In Fig. 4, a speed sensor 51, an acceleration sensor 52, an exterior camera 53, a Lidar 54, an interior camera 55, and a GNSS receiver 56 are shown as examples of the sensor members. The speed sensor 51 detects the speed of the automobile 2 traveling for transportation. The acceleration sensor 52 detects the acceleration of the automobile 2 traveling for movement. Exterior camera 53 is a camera that captures images of the surroundings outside automobile 2. Exterior camera 53 may be, for example, a 360-degree camera that captures the entire surroundings outside automobile 2, or may capture the surroundings outside automobile 2 in a divided manner using multiple cameras. The Lidar 54 is a sensor that uses signals such as infrared rays to detect the surroundings outside the automobile 2. The Lidar 54 generates spatial information outside the vehicle based on reflected infrared waves. 1, and generates the current position and current time of the automobile 2 in which the GNSS receiver 56 is installed. The GNSS satellites 110 may include zenith satellites. The GNSS receiver 56 may also be capable of receiving radio waves from transmitters fixedly installed on the ground.

[0029] In-vehicle camera 55 is a camera that captures images of the interior of automobile 2 in which passengers such as the driver are riding. Here, in-vehicle camera 55 may be a wide-angle camera that can capture a single image of the driver and assistant passenger riding in automobile 2. Such a wide-angle camera that captures images of multiple passengers riding in automobile 2 may be provided in the center of automobile 2 in the vehicle width direction, for example, in the center of the dashboard.

[0030] The detection control device 26 then outputs the detection information of these sensor members and information generated based on the detection information to other control devices via the vehicle network 30. For example, the detection control device 26 may record information on the captured image of the occupant's face taken by the in-vehicle camera 55 in advance in the memory 64, and may identify the occupant riding in the automobile 2 by comparing the captured image with the current captured image of the in-vehicle camera 55. In this case, the detection control device 26 may output information on the identified occupant to another control device through the vehicle network 30 as information generated based on the detection information. The detection control device 26 may also repeatedly obtain the latest captured image of the in-vehicle camera 55 to monitor the condition of the occupant. The occupant may develop a disease while driving. When the detection control device 26 detects such a change in the condition of the occupant, it may transmit an emergency response request from the communication control device 27 to the server device 5. Such a detection control device 26 functions as an occupant monitoring control device.

[0031] FIG. 5 is a configuration diagram of the various control devices 60 in FIG. For example, the communication control device 27 in FIG. 4 may have the configuration shown in FIG. The control device 60 in FIG. 5 includes an in-vehicle communication device 61, an input / output port 62, a timer 63, a memory 64, a CPU 65, and an internal bus 66 to which these are connected.

[0032] The in-vehicle communication device 61 is connected to the vehicle network 30. The in-vehicle communication device 61 inputs and outputs information to and from the in-vehicle communication devices 61 of other control devices through the vehicle network 30. The input / output port 62 is connected to various members connected to the control device 60. In the case of the communication control device 27, a communication device for communication with the base station 3 may be connected. The timer 63 measures time or hour. The time of the timer 63 may be calibrated by the current time acquired by the GNSS receiver 56. This allows the time of the server device 5 and the time of the automobile 2 to match with high accuracy. The memory 64 records various information and programs executed by the CPU 65. The memory 64 may be composed of, for example, a semiconductor memory, a HDD, etc. The semiconductor memory includes, for example, a volatile memory such as a RAM, and a non-volatile memory such as a ROM or an SSD. The CPU 65 reads and executes the programs recorded in the memory 64. In this way, the CPU 65 functions as a vehicle control unit that controls the overall operation of the control device 60 in which the CPU 65 is provided.

[0033] FIG. 6 is a flowchart of the control of transmission of information for emergency response services by the control system 20 of the automobile 2 of FIG. Here, the description will be given assuming that the CPU 65 of the communication control device 27 executes the transmission control of Fig. 6. The CPUs 65 of the various control devices 60 included in the control system 20 of the automobile 2 of Fig. 4 may execute the transmission control of Fig. 6. Also, a plurality of CPUs 65 of the control system 20 may cooperate to execute the transmission control of Fig. 6. Then, the CPU 65 of the communication control device 27 repeatedly executes the transmission control of Fig. 6, for example, when an occupant is in the automobile 2.

[0034] In step ST11, the CPU 65 collects the latest vehicle information of the automobile 2. Here, the vehicle information may be any information that can be collected from each part of the automobile 2. The vehicle information may include information indicating the running state or malfunction of the automobile 2, information indicating the state or malfunction of the occupants, and the like. Information indicating the state of the occupants may be, for example, an image captured by the in-vehicle camera 55. The image captured by the in-vehicle camera 55 may include images of multiple occupants, such as the driver and assistant, who are in the automobile 2.

[0035] In step ST12, the CPU 65 judges whether or not the vehicle status that can be determined based on the vehicle information requires communication with the server device 5. If there is a malfunction of the automobile 2 or an occupant, if an accident of the automobile 2 occurs, or if an occupant requests communication by operating a button (not shown), the CPU 65 may judge that communication with the server device 5 is required based on such information. In this case, the CPU 65 advances the process to step ST13 to execute communication with the server device 5. If it is not judged that communication with the server device 5 is required, the CPU 65 ends this control.

[0036] From step ST13, the CPU 65 starts processing for executing communication with the server device 5. The CPU 65 first selects information to be transmitted from the latest collected vehicle information. If there is a malfunction in the automobile 2, the CPU 65 may select various information about the automobile 2 as the information to be transmitted. If there is a malfunction in an occupant or if the occupant is operating a button, the CPU 65 may select various information about the occupant as the information to be transmitted. If an accident of the automobile 2 has occurred, the CPU 65 may select various information about the automobile 2 and the occupant as the information to be transmitted. Then, when selecting various information about the occupant, the CPU 65 may select an image captured by the in-car camera 55 including the face of the occupant.

[0037] In step ST14, the CPU 65 judges whether or not a facial image of an occupant has been selected as information to be transmitted. If, for example, an image captured by the in-vehicle camera 55 including the face of the occupant has been selected as information to be transmitted, the CPU 65 judges that an image of the occupant's face has been selected, and proceeds to step ST15. If an image captured by the in-vehicle camera 55 including the face of the occupant has not been selected as information to be transmitted, the CPU 65 skips step ST15 and proceeds to step ST16.

[0038] In step ST15, the CPU 65 generates a processed image reflecting the facial image of the occupant based on the image including the face of the occupant selected as the transmission information. This processing control will be described later. As a result, the CPU 65 generates a processed facial image reflecting the facial image of the occupant based on, for example, an image captured by the in-vehicle camera 55 including the face of the occupant selected as the transmission information. Then, the CPU 65 changes the facial image of the occupant to be transmitted from the image captured by the in-vehicle camera 55 to the processed facial image.

[0039] In step ST16, the CPU 65 transmits the transmission information from the communication control device 27 to the server device 5 via the base station 3. If an edited image reflecting the facial image of the occupant has been generated in step ST15, the image captured by the in-vehicle camera 55 is not transmitted. As a result, the information transmitted from the automobile 2 to the server device 5 does not include the facial image of the occupant actually riding in the automobile 2. The facial image of the occupant actually riding in the automobile 2 is not transmitted to the server device 5 via the communication network 6, and can be protected as personal information. Furthermore, in the server device 5, the server communication device 12 receives the transmission information from the automobile 2. The server CPU 18 of the server device 5 executes the processes of steps ST3 to ST8 in FIG. 2. As a result, a face image based on the processed image is displayed on the server display device 13. The operator observes and judges the condition of the occupant of the automobile 2 based on the displayed face image based on the processed image. Furthermore, the operator can establish a communication path with the automobile 2 as necessary to talk directly with the occupant. The operator can make an excellent judgment about the condition of the automobile 2 and the occupant, and provide appropriate vehicle services.

[0040] FIG. 7 is a detailed flowchart of the processed image generation control in step ST15 in FIG.

[0041] In step ST61, the CPU 65 cuts out a face image, which is an image capturing region of the face of the occupant, from the captured image of the in-vehicle camera 55 that has already been acquired as vehicle information acquired from each part of the automobile 2. Here, the facial image of the occupant refers to an image that contains image components of the occupant's face.

[0042] In step ST62, the CPU 65 generates point cloud information indicating the face of the occupant for the cut-out facial image of the occupant. Here, the point cloud information includes a plurality of points indicating the position of the contour of the occupant's face in the facial image of the occupant, and a plurality of points indicating the positions of elements of the occupant's face, such as the eyebrows, eyes, nose, and mouth. In addition to this, for example, the point cloud information may include a plurality of points indicating the position of the contour of the entire head including the head of the occupant. In addition, the plurality of points indicating the positions of the facial elements may be a combination that can indicate the shape and range of the element. For example, the plurality of points for the eyebrows may be a combination that indicates the positions of both ends of each eyebrow and the position between them. The plurality of points for the eyes may be a combination that indicates the positions of the temples, the positions of the tails of the eyes, the positions of the upper eyelids, and the positions of the lower eyelids. The plurality of points for the nose may be a combination that indicates the upper end and lower edges, both left and right edges, and the tip of the nose. The plurality of points for the mouth may be a combination that indicates the outer periphery of the upper lip and the outer periphery of the lower lip. Point cloud information of a plurality of points on the face of these occupants can be generated by a landmark detection process.

[0043] In step ST63, the CPU 65 generates patch information for dividing the face image into a plurality of parts. At this time, the CPU 65 may generate the patch information for dividing the face image by using the point cloud information generated in step ST62 as a criterion for image division. The CPU 65 may generate a plurality of pieces of patch information for the face image by, for example, the Delaunay method. In the Delaunay method, the face image is divided into a plurality of parts so as to obtain triangular patch areas.

[0044] In step ST64, the CPU 65 generates a luminance value for the facial image of the occupant. The CPU 65 may generate an average luminance value, a representative luminance value, or the like for the facial portion of the facial image of the occupant.

[0045] In step ST65, the CPU 65 acquires a plurality of sample face images of a plurality of other people to be used in place of the face of the occupant. Here, the other people may be people different from the occupant involved in the processing, and preferably people unrelated to the occupant. Examples of face images of unrelated people include a face image synthesized from face images of a plurality of real people, and a face image of a non-existent person mechanically generated. The plurality of sample face images are preferably of a plurality of other people who basically have different skin colors and brightness values. The range of selection is wider when the plurality of sample face images are different than when they are similar. It is preferable that the sample face image acquired here is one to which the point cloud information, patch information, and brightness value generated for the face image of the occupant in steps ST62 to ST64 have already been associated. However, if this information has not been associated, the CPU 65 may execute the processes in steps ST62 to ST64 for the acquired sample face image as well to generate the point cloud information, patch information, and brightness value for the sample face image.

[0046] In step ST66, the CPU 65 determines the similarity between the facial image of the occupant and the plurality of sample facial images acquired. For example, the CPU 65 may compare the brightness value determined for the facial image of the occupant with the brightness values ​​of the plurality of sample facial images acquired to determine the similarity for each of the plurality of sample facial images. Here, the CPU 65 may determine the similarity by comparing the brightness values ​​determined for the entire face portion.

[0047] In step ST67, the CPU 65 selects one sample face image to be used for processing from the acquired sample face images. For example, the CPU 65 may select the sample face image having the smallest difference in luminance value with respect to the face image of the occupant and the highest similarity as the sample face image to be used for processing.

[0048] In step ST68, the CPU 65 executes a morphing process on the sample face image to generate a morphed face image for the sample face image. At this time, the CPU 65 executes the morphing process on the sample face image so that the point cloud information and patch information of the sample face image approach the point cloud information and patch information generated for the occupant. After that, the CPU 65 ends this control and returns the process to Fig. 6. If a morphed face image that has been morphed to resemble the face of the occupant is obtained by the process of Fig. 6, it is set as the face image of the occupant to be transmitted. In step ST16, the CPU 65 transmits the face image of another person that has been morphed by the morphing process to the server device 5 instead of the face image of the occupant of the automobile 2. The CPU 65 may transmit to the server device 5 an image in which the image area of ​​the face of the occupant in the image captured by the in-vehicle camera 55 has been replaced with a morphed face image processed by morphing processing.

[0049] FIG. 8 is an explanatory diagram of a facial image processing procedure in the first embodiment. 8 shows an automobile 2 and a server device 5. Time flows from top to bottom.

[0050] In this embodiment, after an occupant image acquisition process (not shown), the CPU 65 of the automobile 2 executes a cutout process, a point cloud generation process, a patch generation process, a process for acquiring a plurality of sample face images, a similarity determination process, a process for selecting one sample face image, and a morphing process in steps ST71 to ST77. Then, in step ST78, the CPU 65 transmits the morphed face image 100 to the server device 5 as a face image of an occupant of the automobile 2. In this case, the server CPU 18 of the server device 5 transmits a plurality of sample face images to the automobile 2 in step ST79. Furthermore, the server CPU 18 of the server device 5 outputs the morphed face image 100 obtained by the morphing process and received from the automobile 2 as a face image of an occupant of the automobile 2 from the server display device 13 in step ST80. This enables the CPU 65 of the automobile 2 to select one of the multiple sample face images acquired from the server device 5, generate the morphed face image 100, and transmit it to the server device 5.

[0051] Unlike the processing procedure of this embodiment, the CPU 65 of the automobile 2 may execute part of the processing from step ST71 to step ST77 and transmit the processed information, and the remaining processing may be executed by the server CPU 18 of the server device 5. Even in this case, by not transmitting the facial image of the occupant itself from the automobile 2 to the server device 5, it becomes difficult for personal information to be transmitted outside the automobile 2.

[0052] Next, the process of controlling the generation of processed images of the faces of the passengers of the automobile 2 will be described with reference to a specific example.

[0053] FIG. 9 is an explanatory diagram of an example of an image 70 captured by the in-vehicle camera 55 of FIG. Since the in-vehicle camera 55 in Fig. 4 is a wide-angle camera, the captured image 70 of the in-vehicle camera 55 in Fig. 9 captures images of multiple occupants, including the driver and assistant driver, who are in the automobile 2. Even if the head of each occupant moves within the vehicle, the entire head of each occupant is included in the captured image 70. In this case, in step ST61 in FIG. 7, the CPU 65 cuts out, for example, an imaging area including the driver's face, which is indicated by a dashed frame in the figure, as a face image 71 of the driver.

[0054] FIG. 10 is an explanatory diagram of an example of a face image 71 which is an image capture region of the driver's face cut out from the captured image 70 of FIG. In the driver's face image 71 of FIG. 10, the entire head of the driver is included without being cut off by the outer periphery of the image. The CPU 65 may cut out the face image 71 of the driver so as to include the driver's eyebrows to the chin, rather than the entire head of the driver. Even in this case, the face image 71 of the driver includes image components of the driver's face. The CPU 65 can generate point cloud information on the elements of the driver's face from the eyebrows to the chin, as described later. However, even when the head portion is cut off in this way, the CPU 65 cuts out the driver's face so that a predetermined width of margins 77 is included on the left and right of the area of ​​the driver's face in the driver's face image 71. The margins 77 are useful in the processing described later.

[0055] FIG. 11 is an explanatory diagram of an example of point cloud information generated for the face image 71 of the driver in FIG. The CPU 65 can generate point cloud information as shown in FIG. 11 for the driver's face image 71 in FIG. 10 in step ST62 in FIG. The point cloud information in FIG. 11 includes a plurality of points 72 on the face itself, such as a plurality of points indicating the position of the driver's facial contour and a plurality of points indicating the positions of elements of the driver's face, such as the eyebrows, eyes, nose, and mouth, as well as a plurality of points 73 on the outer periphery, which is the contour of the facial image itself. Here, a plurality of points 73 on the outer periphery of the image are provided at each corner and the center of each side of the outer periphery of the rectangular image. The plurality of points 73 on the outer periphery of the image are separated from the plurality of points 72 on the driver's face itself due to the inclusion of white space 77 on the left and right of the image 71 of the driver's face. Furthermore, the multiple points 73 on the outer periphery of the image are arranged so as to entirely surround the multiple points 72 on the driver's face itself from the outside.

[0056] FIG. 12 is an explanatory diagram of an example of patch information generated for the driver's face image 71 of FIG. In the patch information of FIG. 12, each patch is a triangle. The CPU 65 can generate patch information shown in FIG. 12 for the driver's face image 71 shown in FIG. 10 by the Delaunay method in step ST63 shown in FIG. The patch information in Fig. 12 includes a patch 84 for the driver's face itself as well as a patch 85 outside the driver's face. The CPU 65 can divide the entire face image into a plurality of triangular patches as shown in Fig. 12 by processing the Delaunay method using a plurality of points 72 for the driver's face itself as well as a plurality of points 73 arranged on the outer periphery of the image. Due to the presence of margins 77 on the left and right of the driver's face image 71, the CPU 65 can divide the entire face image into a plurality of triangular patches without affecting the division of the face itself into a plurality of patches. In contrast, if there is no margin 77 on the left and right of the driver's face image 71, the CPU 65 is likely to divide the face image into a plurality of patches using the Delaunay method in a manner different from that shown in FIG. 12. For example, there is a high possibility that the CPU 65 will divide the face into patches that are not suitable for face patch information, such as processing points on the left and right outer edges of the image as part of the face contour. In order to avoid such a situation, in this embodiment, the face image is cut out so that the width of the face is about 1 / 3 of the width of the face image, as shown in FIG. 10 and FIG. 13 described later. For example, by providing margins on the left and right sides of the face with a width of at least 25% or more of the width of the face image, the CPU 65 can generate suitable face patch information as shown in FIG. 12 using the Delaunay method. The CPU 65 may cut out an imaging area that is larger than the driver's face in at least the width direction of the driver's face, as the imaging area of ​​the driver's face.

[0057] In this way, in the cutout process, the CPU 65 cuts out an image capture area that is larger than the occupant's face at least in the width direction of the face from the captured image 70 acquired from the in-vehicle camera 55, and generates a face image as the image capture area of ​​the occupant's face. Then, in the point cloud generation process, the CPU 65 generates a plurality of point cloud information about the outer periphery of the cut-out image capture area of ​​the occupant's face, including a position away from the occupant's face. In addition, in the patch generation process, the CPU 65 uses the face point cloud information indicating the contour of the face and the positions of the facial elements as well as the plurality of point cloud information on the outer periphery of the cut-out image capture area of ​​the occupant's face as a criterion for image division to generate patch information for dividing the cut-out image capture area of ​​the occupant's face into units based on triangles.

[0058] Next, the information of the sample face image 91 that the CPU 65 acquires in step ST65 of FIG. 17 will be described with reference to FIG. 13 to FIG. FIG. 13 is an explanatory diagram of an example of a combination of a plurality of sample face images 91 and 92. In FIG. Fig. 13 shows a face image group 90 acquired by the CPU 65. The face image group 90 in Fig. 13 includes a sample face image 91 of a first other person who is fair-skinned, and a sample face image 91 of a second other person who is dark-skinned. The luminance value of the sample face image 91 of the first other person and the luminance value of the sample face image 91 of the second other person who is dark-skinned are different from each other. Then, the CPU 65 selects, from the face image group 90 in FIG. 13, a first sample face image 91 of another person whose difference in luminance value with the face image 71 of the driver in FIG. 10 is small, based on a similarity judgment based on the luminance value.

[0059] FIG. 14 is an explanatory diagram of an example of point cloud information generated for the first sample face image 91 of another person selected in FIG. FIG. 15 is an explanatory diagram of an example of patch information generated for the sample face image 91 of the first other person selected in FIG.

[0060] In the first sample face image 91 of another person in FIG. 13, the whole head of another person is included without being cut off by the outer periphery of the image so that there is a margin around the head. In this case, the CPU 65 can generate point cloud information for the first other's sample face image 91 as shown in Fig. 14 by performing point cloud generation processing for the first other's sample face image 91. In this case, the point cloud information for the first other's sample face image 91 includes a plurality of points 82 on the face itself indicating the contour of the face and the positions of the facial elements, as well as a plurality of points 83 on the outer periphery of the face image. Furthermore, the CPU 65 performs patch generation processing on the sample face image 91 of the first other person to generate patch information for dividing the sample face image 91 of the first other person into units based on triangles, as shown in Fig. 15. The patch information includes a patch 84 for dividing the face of the first other person itself, and a patch 85 for dividing the outside of the face of the first other person. 10, the CPU 65 may generate the point cloud information in FIG. 14 and the patch information in FIG. 15 for the sample face image 91 of the first other person in FIG. 13.

[0061] 13 to 15 may be recorded in the memory 64 as information of the image group 90 before the processing of step ST65 in Fig. 17. Furthermore, the CPU 65 may obtain information of the face image group 90 including information of the sample face image 91 of the first other person in Fig. 13 to 15 from the server device 5 in the processing of step ST65 in Fig. 17. Furthermore, the information of the face image group 90 recorded in the memory 64 may be obtained in advance from the server device 5.

[0062] 16 is an explanatory diagram of an example of a morphed face image 100 transmitted from the automobile 2 to the outside as a face image of an occupant. This morphed face image 100 is an image transmitted from the automobile 2 to the outside in place of the face image 71 of the driver. The CPU 65 executes a morphing process on the first sample face image 91 of another person in step ST68 of FIG. In the morphing process, the CPU 65 uses the information in FIGS. 13 to 15 regarding the sample face image 91 and the information in FIGS. 10 to 12 regarding the driver's face image 71. A plurality of points 72 and 73 included in the point cloud information of the driver's face image 71 in FIG. 11 correspond to a plurality of points 82 and 83 included in the point cloud information of the sample face image 91 in FIG. 15. Furthermore, the multiple patches 74 and 75 included in the patch information of the driver's face image 71 in FIG. 12 correspond to the multiple patches 84 and 85 included in the patch information of the sample face image 91 in FIG. In this way, the point cloud information and patch information of the sample face image 91 are generated so as to correspond well to the point cloud information and patch information of the driver's face image 71 .

[0063] In this case, the CPU 65, for example, moves the positions of the points 82, 83 in the point cloud information of the sample face image 91 closer to the positions of the corresponding points 72, 73 in the point cloud information of the driver's face image 71. As a result, the positions and ranges of the patches 84, 85 in the patch information of the sample face image 91 change so as to overlap with the positions and ranges of the patches 74, 75 in the patch information of the driver's face image 71. As a result, the face of the other person in the first sample face image 91 of FIG. 13 approaches the face of the driver in the image 71 of FIG.

[0064] In this way, the CPU 65 executes a morphing process on the sample face image 91 so that the point cloud information and patch information of the sample face image 91 of another person become closer to the point cloud information and patch information of the face image 71 of the driver. Here, when the positions of the points 82, 83 in the point cloud information of the sample face image 91 overlap with the positions of the points 72, 73 in the point cloud information of the driver's face image 71, the sample face image 91 of the first other person is morphed at a rate of 100%. The facial contour, the position and the size of the facial elements in the sample face image 91 can be approximately the same as those in the actual driver's face image 71. However, since the sample face image 91 used in the morphing process is not the driver himself but a different person, even if the morphing process is performed at a rate of 100%, the morphed face image 100 will not be the driver's face image 71 itself. When the morphing rate is 50%, the morphed face image 100 will be intermediate between the driver's face in the image 71 of FIG. 10 and the face of the other person in the sample image 81 of FIG. 13, as shown in FIG. 16. Even in this case, the morphed face image 100 contains the driver's facial features and expressions at a morphed ratio. The CPU 65 executes the morphing process by setting an arbitrary value as the morphing ratio for the sample face image 91. The value of the morphing ratio may be fixed, or may be arbitrarily set by the driver or the like.

[0065] The CPU 65 transmits the morphed face image 100 generated by this morphing process to the server device 5 in place of the captured image 70 of the in-car camera 55 and in place of the face image of the occupant of the automobile 2. The CPU 65 can protect the face image 71 of the driver as personal information and prevent it from being transmitted outside the vehicle.

[0066] As described above, in the vehicle service providing system 1 of the present embodiment, the CPU 65 of the automobile 2 executes the passenger image acquisition process, the brightness value generation process, the sample image acquisition process, the determination process, the selection process, and the morphing process. Then, the server CPU 18 outputs the morphed face image 100 by the morphing process from the server display device 13 instead of the facial image of the vehicle passenger. As a result, a facial image reflecting the state of the passenger, such as the facial expression, can be output to the server display device 13. As a result, the operator can grasp the current actual state and facial expression of the passenger based on the output facial image. On the other hand, if the image components of the passenger's face are abstracted, replaced, or masked, it is difficult for the operator to accurately grasp the current actual state and facial expression of the passenger in the displayed image. In particular, in this embodiment, instead of performing a morphing process to bring a single fixed face image of another person different from the occupant closer to the face of the occupant, a face image having a high similarity to the luminance value of the occupant's face is selected from a face image group 90 consisting of a plurality of sample face images of other people. As a result, the face image of the occupant of the automobile 2 output from the server display device 13 of the server device 5 becomes a face image having a luminance value close to that of the occupant's face depending on the occupant. Then, by selecting from a combination of a plurality of sample face images 91, 92 having different luminance values, it is possible to select a sample face image 91 that is close to the actual mood of the occupant. As a result, the operator can grasp the mood of the occupant and judge the state of the occupant more accurately based on the output of a different morphed face image 100 depending on the occupant, unlike the case where a single fixed face image of another person is always output. Furthermore, the CPU 65 of the automobile 2 executes occupant image acquisition processing, extraction processing, point cloud generation processing, patch generation processing, sample image acquisition processing, and morphing processing, and transmits the information generated by the processing to the server device 5 in place of the occupant's facial image. As a result, in this embodiment, it is possible to prevent the occupant's facial image 71 itself, which needs to be protected as the occupant's personal information, from being transmitted to the outside. In this manner, in this embodiment, by selecting and morphing a sample face image 91 that is similar to the brightness value of the actual occupant's face, it is possible to protect personal information while minimizing the loss of convenience of the vehicle service providing system 1.

[0067] Moreover, in this embodiment, the CPU 65 of the automobile 2 further executes a cutting process for cutting out a face image 71, which is an image capture area of ​​the occupant's face, from the captured image 70 acquired from the in-vehicle camera 55, a point cloud generation process for generating face point cloud information indicating the positions of the facial contour and facial elements in the cut-out face image 71 of the occupant by a landmark detection process, and a patch generation process for generating patch information for dividing the cut-out image capture area of ​​the occupant's face into units based on triangles, using the generated point cloud information as a criterion for image division. Then, in the morphing process, the CPU 65 executes a morphing process for bringing the point cloud information and patch information for the sample face image 91 selected by the selection process closer to the point cloud information and patch information generated for the occupant, thereby generating a morphed face image 100. The morphed face image 100 is closer to the face of the occupant not only in terms of its brightness value but also in terms of the shape of the face. This makes it possible to generate and output a facial image that reflects not only the luminance value of the occupant's face but also the state of the occupant's facial expression, etc., as a morphed facial image 100 corresponding to the occupant's face. As a result, the operator can accurately grasp the current actual state and expression, etc., of the occupant based on the output facial image 100.

[0068] [Second embodiment] Next, a vehicle service providing system 1 according to a second embodiment of the present invention will be described. In the vehicle service providing system 1 of this embodiment, the processed image generation control is different from that of the above-described embodiment. The same configurations and processes as those in the above-described embodiment are designated by the same reference numerals as those in the above-described embodiment. The following mainly describes the differences from the above-described embodiment.

[0069] FIG. 17 is a detailed flowchart of the processed image generation control in the second embodiment of the present invention. The CPU 65 of the communication control device 27 provided in the control system 20 of the automobile 2 may execute the processed image generation control of FIG. 17 in step ST15 when the transmission control of FIG. 6 is repeatedly executed while passengers are aboard the automobile 2. Steps ST61 to ST63 in Fig. 17 are the same as those in Fig. 7. After step ST63, the CPU 65 advances the process to step ST91.

[0070] In step ST91, the CPU 65 generates a luminance value for each patch of the facial image of the occupant. The CPU 65 may generate a luminance value for each patch of the image area of ​​each of the multiple patches included in the patch information.

[0071] In step ST92, the CPU 65 judges whether or not there is a patch with a peculiar luminance value in the luminance values ​​generated for each patch of the facial image of the occupant in step ST91. Here, the peculiar luminance value may be, for example, a value that differs from the average luminance value of the facial image of the occupant by a predetermined threshold value or more. In this way, for example, if the cheek of the occupant's face is scratched or bloody, the CPU 65 can judge the patch corresponding to that part as a patch with a peculiar luminance value. On the other hand, the luminance value of each part of the face differs for each facial element. For example, the luminance value of the eyes and the luminance value of the mouth may be significantly different from the average luminance value of the whole face. In order not to determine a patch of such a part as a patch of an unusual luminance value, the CPU 65 may determine the attribute information of each patch for the part of the face to which each patch corresponds. Then, for a patch determined to be a part with an inherently unusual luminance value, it may be determined whether or not it is a patch of an unusual luminance value according to a determination criterion specific to each patch. Furthermore, for example, to identify and monitor the occupant, facial images of the occupant may be pre-recorded in the memory 64. In this case, the CPU 65 may compare the luminance value generated for each patch of the facial image of the occupant in step ST91 with the facial image pre-recorded for the occupant in the memory 64, and determine that the patch having a difference equal to or greater than a threshold value is a patch having a unique luminance value. This enables the CPU 65 to determine, as a determination process, whether or not a patch area defined by the patch information generated in the patch generation process includes a peculiar luminance value indicating an injury to an occupant. If there is a patch with a peculiar luminance value in the facial image of the occupant, the CPU 65 advances the process to step ST93. On the other hand, if there is no patch with a unique brightness value in the facial image of the occupant, the CPU 65 advances the process to step ST65. In this case, the CPU 65 executes the processes of steps ST65 to ST68 to generate a morphed facial image, similarly to FIG. After that, the CPU 65 ends this control and returns the process to Fig. 6. If a morphed face image 100 that has been morphed to resemble the face of the occupant is obtained by the process of Fig. 6, it is set as the face image of the occupant to be transmitted. In step ST16, the CPU 65 transmits the morphed face image 100 that has been processed by the morphing process to the server device 5 instead of the face image of the occupant of the automobile 2. The CPU 65 may transmit to the server device 5 an image in which the image area of ​​the face of the occupant in the image captured by the in-vehicle camera 55 has been replaced with a morphed face image 100 obtained by morphing processing.

[0072] In step ST93, the CPU 65 obtains a plurality of sample face images to be used in place of the faces of the occupants, similarly to step ST65.

[0073] In step ST94, the CPU 65 determines the degree of similarity between the facial image of the occupant and the plurality of acquired sample facial images, similarly to step ST66.

[0074] In step ST95, the CPU 65 selects one sample face image to be used for processing from the plurality of acquired sample face images, similarly to step ST67.

[0075] In step ST96, the CPU 65 changes the patch corresponding to the unique luminance value identified in step ST92 for the occupant's face image among the multiple patches for the selected sample face image to the identified unique luminance value. Thereby, the CPU 65 can change, as a modification process, the luminance value of the patch area corresponding to the determination of the determination process in the selected sample face image to a peculiar luminance value indicating injury to an occupant. The CPU 65 may generate a selected sample face image that includes patches of unique brightness values ​​similar to those of the occupant's face image. After that, the CPU 65 advances the process to step ST68. In the morphing process of step ST68, the CPU 65 executes a morphing process to bring the point cloud information and patch information of the sample face image after the change process closer to the point cloud information and patch information generated for the occupant, thereby enabling the CPU 65 to generate a morphed face image that includes a unique luminance value similar to that of the face image of the occupant. Then, the CPU 65 ends this control and returns the process to Fig. 6. If a morphed face image obtained by the process of Fig. 6 has been morphed to resemble the face of the occupant, the morphed face image is set as the face image of the occupant to be transmitted. In step ST16, the CPU 65 transmits the morphed face image obtained by the morphing process to the server device 5 instead of the face image of the occupant of the automobile 2.

[0076] Next, the process of controlling the generation of processed images of the faces of the passengers of the automobile 2 will be described with reference to a specific example.

[0077] FIG. 18 is an explanatory diagram of an example of a face image 71 which is an imaging area of ​​the driver's face in this embodiment. 10, the driver's face image 71 in FIG. 18 may be cut out by the CPU 65 from the captured image 70 in FIG. 10, the driver's face image 71 in FIG. 18 includes an image component of a scar 104 on the driver's right cheek.

[0078] FIG. 19 is an explanatory diagram of an example of point cloud information and patch information generated for the driver's face image 71 of FIG. The point cloud information and patch information generated for the driver's face image 71 in FIG. 19 are basically the same as the point cloud information and patch information generated for the driver's face image 71 in FIG. Then, in step ST91, the CPU 65 generates luminance values ​​for each of the multiple patches generated for the driver's face image 71 in Fig. 19. The luminance value of the patch including the scar 104 on the right cheek, which is shaded in the figure, differs from the luminance values ​​of the surrounding patches due to the luminance value of the scar 104. In addition, the luminance value of the patch including the scar 104 on the right cheek differs greatly from the average luminance value of the entire face. As a result, in step ST92, the CPU 65 determines that the patch including the scar 104 on the right cheek of the driver's face image 71 in Fig. 19 is a patch with a unique luminance value. In this case, the CPU 65 advances the process from step ST93 to step ST96.

[0079] FIG. 20 is an explanatory diagram of an example of point cloud information and patch information generated for a sample face image 91 selected from among a plurality of face images. The point cloud information and patch information generated for the sample face image 91 in FIG. 20 are basically the same as the point cloud information and patch information generated for the sample face image 91 in FIG. Then, in step ST96, the CPU 65 changes the luminance value of the patch corresponding to the patch determined to be unique in step ST92 among the multiple patches of the sample face image 91 to the unique luminance value used in the determination in step ST92. As a result, the luminance value of the patch marked with diagonal lines in the figure will be different from that of the surrounding patches due to the luminance value of the scar 104 in the driver's face image 71.

[0080] FIG. 21 is an explanatory diagram of an example of a morphed face image 100 that is transmitted from the automobile 2 to the outside as a face image of an occupant. After step ST96, the CPU 65 executes a morphing process on the sample face image 91 of FIG. 20 in step ST68. As a result, the CPU 65 generates the morphed face image 100 shown in FIG. The morphing face image 100 in FIG. 21 includes a portion 107 on the right cheek of the face that has a different brightness value from its surroundings. By displaying such an image on the server display device 13, the operator can easily notice that the driver has a scar on his right cheek.

[0081] As described above, in this embodiment, the CPU 65 of the automobile 2 can determine whether or not a patch area defined by the patch information generated in the patch generation process includes a luminance value indicating an injury to an occupant. Then, the CPU 65 of the automobile 2 changes the luminance value of the patch corresponding to the determined patch area in the selected sample face image 91 through a change process to a unique luminance value that can indicate an injury to an occupant. Then, in a morphing process, the CPU 65 of the automobile 2 executes a morphing process to bring the point cloud information and patch information for the sample face image 91 after the change process closer to the point cloud information and patch information generated for the occupant. As a result, the morphed face image 100 output from the server device 2 includes a luminance portion corresponding to the injury of the occupant. The operator can accurately grasp the condition of the occupant, such as the presence or absence of an injury, based on the output "occupant's face image."

[0082] In particular, in this embodiment, the facial images of other people are selected based on the luminance value of the facial image 71 of the driver as a criterion. As a result, the luminance portion corresponding to the injury of the occupant tends to appear clearly in the facial image of another person as well as in the facial image 71 of the driver.

[0083] The above-described embodiment is an example of a preferred embodiment of the present invention, but the present invention is not limited to this, and various modifications and changes are possible without departing from the gist of the invention.

[0084] In the embodiment described above, the CPU 65 cuts out a part of the image captured by the wide-angle in-car camera 55 as the face image 71 including the face of the occupant, and performs subsequent processing on the cut-out image. Alternatively, for example, the CPU 65 may use the image captured by the wide-angle in-vehicle camera 55 as the facial image of the occupant as is for subsequent processing. In this case, as shown in Fig. 9, for example, a person other than the occupant to be morphed is included in the facial image 71. The CPU 65 may also morph this person. Alternatively, for example, the CPU 65 may abstract the person, simply replace it with another image, or perform a mask process on the person. [Explanation of symbols]

[0085] 1...vehicle service providing system, 2...automobile (vehicle), 3...base station, 4...communication network, 5...server device, 6...communication network, 9...road service, 11...server GNSS receiver, 12...server communication device, 13...server display device (output section), 14...server operation device, 15...server voice device, 16...server timer, 17...server memory, 18...server CPU, 19...server bus, 20...control system, 21...drive control device, 22...steering control device, 23...braking control device, 24...driving control device, 25...driving operation control device, 26...detection control device, 27...communication control device, 28...call control device, 30...vehicle network, 31...central gateway device, 32...bus cable, 4 1...Steering wheel, 42...Brake pedal, 43...Accelerator pedal, 44...Shift lever, 46...Speaker, 47...Microphone, 51...Speed ​​sensor, 52...Acceleration sensor, 53...External camera, 54...Lidar, 55...In-vehicle camera, 56...GNSS receiver, 60...Control device, 61...In-vehicle communication device, 62...Input / output port, 63...Timer, 64...Memory, 65...CPU, 66...Internal bus, 70...Captured image, 71...Occupant's facial image, 72,73...Point, 74,75...Patch, 77...Margin, 82,83...Point, 84,85...Patch, 90...Facial image group, 91,92...Sample facial image, 100...Morphing facial image (image that can be transmitted from a vehicle as an occupant's facial image), 110...GNSS satellite

Claims

1. A system for transmitting information from a vehicle to a server device and providing a service based on the information received by the server device, comprising: The vehicle includes an in-vehicle camera capable of capturing an image of an occupant of the vehicle and a vehicle control unit, and the server device includes an image output unit and a server control unit, The vehicle control unit includes: An occupant image acquisition process for acquiring an image including a face of an occupant from the in-vehicle camera; A luminance value generation process for generating a luminance value of the face of the occupant in the captured image; A sample image acquisition process for acquiring a plurality of sample face images of a plurality of people other than the passenger, which are cut out from the captured image; a determination process for determining a similarity between the luminance values ​​of the acquired plurality of sample face images and the luminance value of the face of the occupant generated by the luminance value generation process; a selection process for selecting a sample face image having the highest similarity in luminance value in the determination process; a morphing process for performing a morphing process on the selected sample face image so as to approximate the face of the occupant in the captured image to generate a morphed face image; and transmitting the morphed face image generated by the process to the server device in place of the captured image including the face of the occupant; The server control unit Using information received from the vehicle control unit of the vehicle, it is possible to execute processes among the brightness value generation process, the sample image acquisition process, the determination process, the selection process, and the morphing process that are not executed by the vehicle control unit, and output the morphed face image obtained by the morphing process from the output unit in place of an image of the face of an occupant of the vehicle. Vehicle service provision system.

2. The vehicle control unit includes: execute the luminance value generation process, the sample image acquisition process, the determination process, the selection process, and the morphing process together with the occupant image acquisition process, and transmit the morphed face image obtained by the morphing process to the server device instead of the face image of the vehicle occupant. The vehicle service providing system according to claim 1 .

3. The vehicle control unit includes: In the sample image acquisition process, a plurality of the sample face images having different luminance values ​​are acquired from a memory of the vehicle or the server device. The vehicle service providing system according to claim 2 .

4. The vehicle control unit further includes: A cutting process for cutting out an image capturing area of ​​a face of an occupant from the captured image acquired by the in-vehicle camera; A point cloud generation process for generating face point cloud information indicating the positions of the facial contour and facial elements in the captured image area of ​​the cut-out face of the occupant; A patch generation process is executed to generate patch information for dividing the captured image area of ​​the face of the occupant by using the generated point cloud information as a criterion for dividing the image. In the morphing process, a morphing process is performed to approximate the point cloud information and the patch information for the sample face image selected by the selection process to the point cloud information and the patch information generated for the occupant, thereby generating a morphed face image. The vehicle service providing system according to any one of claims 1 to 3.

5. The vehicle control unit further includes: a determination process for determining whether a patch area defined by the patch information generated in the patch generation process includes a peculiar luminance value indicating an injury to an occupant; A modification process is executed to modify the luminance value of a patch area corresponding to the determination of the determination process in the selected sample face image to a specific luminance value indicating an injury to an occupant; In the morphing process, a morphing process is performed to approximate the point cloud information and the patch information for the sample face image after the change process to the point cloud information and the patch information generated for the occupant, thereby generating a morphed face image. The vehicle service providing system according to claim 4.

Citation Information

Patent Citations

  • Image processing apparatus, image processing system, image processing method, and computer program

    JP2014067131A

  • Information processing device, mobile device, and information process system, method and program

    JP2021043571A

  • System and method for processing video to provide facial de-identification

    US9799096B1

  • Autonomous vehicle system

    WO2020205655A1

Cited By

  • Apparatus and method for detecting physical condition of vehicle occupant

    WO2026212113A1