Vehicle service provision system

Abstract

To provide a vehicle service providing system which protects personal information so as to reduce convenience as little as possible.SOLUTION: In the vehicle service providing system in which a vehicle and a server device transmit and receive information to and from each other via a communication network, a vehicle control unit generates point group information and patch information of a facial image of a passenger captured by an in-vehicle camera and transmits them instead of the facial image of the passenger together with classification information C1 to C3 of the passenger or the vehicle. A server memory records information of a plurality of sample facial images 91 to 94 associated with classification information 95 to 98 of sample facial images. A server control unit uses the received classification information C1 to C3 of the passenger or the vehicle to select a sample facial image from the plurality of sample facial images 91 to 94 and performs morphing processing using the selected sample facial image and outputs a morphed facial image from an output unit instead of the facial image of the passenger of the vehicle.SELECTED DRAWING: Figure 13

Description

Technical Field

[0001] The present invention relates to a vehicle service providing system.

Background Art

[0002] In vehicles such as automobiles, during movement, problems with the vehicle itself or passengers may occur. In addition, it may be desirable for vehicles and passengers to receive various content services during movement. These vehicle services may include emergency response services for dealing with problems with the vehicle and passengers, passenger monitoring services for that purpose, content providing services, and the like. And in such a vehicle service providing system, in order to provide vehicle services, basically, information is transmitted from the vehicle to the server device, and the server device provides services based on the received information.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] Incidentally, in such vehicle service provision systems, especially for the emergency response service and occupant monitoring service mentioned above, it is desirable to transmit images captured by in-vehicle cameras showing the occupant's condition from the vehicle to the server device. By displaying the images captured by the in-vehicle cameras on the server device, operators can accurately grasp the occupant's current condition and facial expressions from the captured images, and as a result, provide the most appropriate service based on the occupant's current condition.

[0005] On the other hand, when transmitting information from a vehicle to a server device in this manner, the transmission of such information may be restricted based on laws and regulations. For example, Europe has its own strict restrictions on the transmission and reception of personally identifiable information. Furthermore, companies that are trying to address the protection of such personal information will want to voluntarily impose reasonable restrictions on the information they transmit from vehicles to server equipment. Furthermore, the images of the vehicle occupants captured as described above include image components of the occupants' faces. The occupants' faces are personal information and should be appropriately protected.

[0006] Patent documents 1 to 3 disclose methods for abstracting, replacing, or masking images of occupants' faces included in captured images. However, if the images of the crew members' faces are abstracted, replaced, or masked, as in Patent Documents 1-3, even if the images are output to the server device, it becomes difficult for the operator to accurately grasp the current state and facial expressions of the crew members.

[0007] Thus, it is desirable for vehicle service provision systems to protect personal information while minimizing any loss of convenience. [Means for solving the problem]

[0008] A vehicle service provision system according to one embodiment of the present invention is a system that transmits information from a vehicle to a server device and provides services based on the information received by the server device, wherein the vehicle has an in-vehicle camera capable of capturing images of the occupants of the vehicle and a vehicle control unit, and the server device has an image output unit, a server memory and a server control unit, and the vehicle control unit performs the following: occupant image acquisition processing to acquire an image including the occupants' faces from the in-vehicle camera; cropping processing to crop out the image region of the occupants' faces from the acquired image; point cloud generation processing to generate point cloud information of faces indicating the contours of the faces and the positions of facial elements in the cropped image region of the occupants' faces; and patch generation processing to generate patch information to divide the cropped image region of the occupants' faces using the generated point cloud information as a criterion for image division, and instead of an image including the occupants' faces, the information of the point cloud information and patch information generated by the processing is used with classification information of the occupants or the vehicle that cannot uniquely identify the occupants. In addition, the system transmits to the server device, and the server memory records information on multiple sample face images, which include point cloud information and patch information as information on face images different from the occupant extracted from the captured image, and which are associated with classification information of the sample face image. The server control unit performs a sample image acquisition process to select a sample face image from among the multiple sample face images recorded in the server memory, using the classification information of the occupant or the vehicle received from the vehicle control unit, to which information corresponding to the classification information of the occupant or the vehicle is associated. It also performs a morphing process to generate a morphed face image by performing a morphing process using the point cloud information and patch information of the selected sample face image and the point cloud information and patch information generated for the occupant. The morphed face image produced by the morphing process is output from the output unit in place of the face image of the occupant of the vehicle. [Effects of the Invention]

[0009] In the vehicle service provision system of the present invention, the vehicle control unit of the vehicle and the server control unit of the server device perform occupant image acquisition processing, cropping processing, point cloud generation processing, patch generation processing, sample face image acquisition processing, and morphing processing. The server control unit then outputs a morphed face image obtained through morphing processing from the output unit in place of the vehicle occupant's face image. As a result, the output unit can output a face image that reflects the occupant's state, such as their facial expression. Consequently, the operator can understand the occupant's current actual state and facial expression based on the output face image. In contrast, if, for example, the crew's facial images are abstracted, replaced, or masked, it becomes difficult for the operator to accurately grasp the crew's current actual state and facial expressions in those images. Furthermore, the vehicle's control unit performs occupant image acquisition processing, cropping processing, point cloud generation processing, and patch generation processing. The vehicle's control unit then transmits the information generated by the processing to the server device in place of the captured image, which includes the occupant's face. As a result, the vehicle of the present invention can avoid transmitting the occupant's face image itself, which needs to be protected as personal information of the occupant, to the outside. Thus, the present invention makes it possible to protect personal information while minimizing any impairment to the convenience of the vehicle service provision system.

[0010] Furthermore, in the present invention, the server device includes a server memory and a server control unit, along with an image output unit. The server memory records information on multiple sample face images, which include point cloud information and patch information as information on face images different from the occupant extracted from the captured image, and which are associated with classification information of the sample face image. The server control unit then selects a sample face image from among the multiple sample face images recorded in the server memory, using classification information of an occupant or vehicle that does not uniquely identify the occupant, which is received from the vehicle control unit, and which is associated with information corresponding to the classification information of the occupant or vehicle. Such a server device can store a large number of sample face images in server memory that can be used effectively for each occupant. Moreover, even with such a large number of sample face images stored in server memory, the server control unit can quickly narrow down and select the sample face images to be used for morphing processing through a simple and low-load process of selection based on occupant or vehicle classification information. The server control unit can quickly select sample face images that can correspond well to the occupant for morphing processing through a less loady process, without performing a high-load process such as comparing and selecting based on the similarity of facial brightness values ​​for all of the numerous sample face images stored in server memory. If other sample face images correspond well to the occupant, the resulting morphed face image is likely to be closer to the occupant than if they did not correspond well. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a system configuration diagram of a vehicle service provision system according to an embodiment of the present invention. [Figure 2] Figure 2 is an explanatory diagram illustrating the basic communication procedure between the vehicle shown in Figure 1 and the server device in an example of providing emergency response services in the vehicle service provision system shown in Figure 1. [Figure 3] Figure 3 is a diagram showing the configuration of the control system of the automobile shown in Figure 1. [Figure 4] Figure 4 is a basic configuration diagram of the communication control device shown in Figure 3. [Figure 5] Figure 5 is a flowchart showing the control of information transmission for emergency response services using the vehicle control system shown in Figure 3. [Figure 6] Figure 6 is a detailed flowchart of the generation control of morphing processing information in step ST15 of Figure 5. [Figure 7] Figure 7 is a diagram showing the configuration of the server device shown in Figure 1. [Figure 8]FIG. 8 is a flowchart of generation control of a processed image for an emergency response service by the server CPU of the server device in FIG. 7. [Figure 9] FIG. 9 is an explanatory diagram of an example of an imaging image by the in-vehicle camera in FIG. 4. [Figure 10] FIG. 10 is an explanatory diagram of an example of a face image that is an imaging area of the driver's face cut out from the imaging image in FIG. 9. [Figure 11] FIG. 11 is an explanatory diagram of an example of point cloud information generated for the driver's face image in FIG. 10. [Figure 12] FIG. 12 is an explanatory diagram of an example of patch information generated for the driver's face image in FIG. 10. [Figure 13] FIG. 13 is an explanatory diagram of an example of a combination of a plurality of sample face images. [Figure 14] FIG. 14 is an explanatory diagram of an example of point cloud information generated for the sample face image selected in FIG. 13. [Figure 15] FIG. 15 is an explanatory diagram of an example of patch information generated for the sample face image selected in FIG. 13. [Figure 16] FIG. 16 is an explanatory diagram of an example of a morphing face image that is displayed and output instead of the passenger's face image.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described based on the drawings.

[0013] FIG. 1 is a configuration diagram of a vehicle service providing system 1 according to an embodiment of the present invention. The vehicle service providing system 1 in FIG. 1 includes an automobile 2 and a server device 5 for providing vehicle services to the automobile 2. The automobile 2 and the server device 5 transmit and receive information to each other through a communication network 6. Here, automobile 2 is just one example of a vehicle, and can basically be any vehicle capable of carrying multiple passengers. In automobile 2, which can carry multiple passengers, the driver and passenger can sit side by side in the width direction of the vehicle. However, the vehicles to which the present invention can be applied are not limited to automobile 2 having such features.

[0014] The communication network 6 in Figure 1 has multiple base stations 3 arranged along the driving path of the vehicle 2, and a communication network 4 to which a server device 5 is connected along with the multiple base stations 3. The base stations 3 may be, for example, base stations of a carrier communication network for mobile terminals, or base stations for ITS services or ADAS services to the vehicle 2. Some fifth-generation base stations of carrier communication networks have high-performance information processing capabilities. In this case, the server device 5 can be distributed across multiple base stations 3.

[0015] During its journey, vehicle 2 may experience malfunctions, either within the vehicle itself or affecting its occupants. In addition, passengers may use various content services while traveling in vehicle 2. These content services include, for example, music streaming services and video streaming services. The vehicle service provision system 1 can provide these vehicle services to the vehicle 2 by utilizing a server device 5 that sends and receives information with the vehicle 2. The vehicle service provision system 1 may provide, for example, emergency response services to address malfunctions in the vehicle 2 or its occupants, and monitoring services for that purpose. The vehicle service provision system 1 may also provide content information from the server device 5.

[0016] Figure 2 is an explanatory diagram illustrating the basic communication procedure between the vehicle 2 and the server device 5 in an example where emergency response services are provided in the vehicle service provision system 1 of Figure 1. Figure 2 shows the automobile 2 and the server device 5. Time flows from top to bottom.

[0017] In Figure 2, in step ST1, the vehicle 2 acquires information about itself and its occupants, and in step ST2, it transmits the acquired information to the server device 5. The information transmitted from the vehicle 2 to the server device 5 may basically include information indicating the status of the vehicle 2, information indicating the status of the occupants, and so on. As shown in step ST3, the server device 5 waits for information to be received from the vehicle 2. When information is received from the vehicle 2, the server device 5 outputs the received information in step ST4, such as by displaying it. This allows the operator to confirm the information of the vehicle 2 and its occupants. If the operator determines that confirmation with the occupants is necessary, the operator operates the server device 5. After outputting the received information, the server device 5 determines in step ST5 whether communication is necessary. Upon operator operation, the server device 5 determines that communication is necessary and starts communication with the vehicle 2 in step ST6. As a result, as shown in step ST7, a communication channel is established between the vehicle 2 and the server device 5, enabling the operator and occupants to communicate by voice or video. Then, if the operator determines through communication with the occupants that an emergency response is necessary, the operator operates the server device 5. The server device 5 makes an emergency call to the road service 9, as shown in step ST8. Based on that call, the road service 9 dispatches a vehicle to the car 2, as shown in step ST9. The road service 9 provides emergency vehicle service. This allows the vehicle 2 or its occupants to enjoy the emergency response services provided by the vehicle service delivery system 1.

[0018] Thus, in the vehicle service provision system 1, in order to provide vehicle services, information is basically transmitted from the vehicle 2 to the server device 5, and the server device 5 provides services based on the information it receives. Incidentally, in such a vehicle service provision system 1, especially in the emergency response service and occupant monitoring service described above, it is desirable to transmit images captured by the in-vehicle camera 55, which shows the occupant's condition, from the vehicle 2 to the server device 5. By displaying the images captured by the in-vehicle camera 55 on the server device 5, the operator can accurately grasp the occupant's current condition and facial expression from the captured images. As a result, the operator can provide the most appropriate service according to the occupant's current condition.

[0019] On the other hand, when transmitting information from the automobile 2 to the server device 5 in this manner, the transmission of such information may be restricted based on laws and regulations. For example, Europe has its own strict restrictions on the transmission and reception of personally identifiable information. Furthermore, companies that are trying to address the protection of such personal information will consider voluntarily restricting the information they transmit from the automobile 2 to the server device 5. Furthermore, the images of the occupants of the aforementioned vehicle 2 include image components of the occupants' faces. The occupants' facial photographs, like their names, are considered personal information and should be appropriately protected as required by law or as necessary.

[0020] Thus, in the vehicle service provision system 1, it is desirable to protect personal information while minimizing any loss of convenience.

[0021] Figure 3 is a diagram showing the configuration of the control system 20 of the automobile 2 shown in Figure 1. The control system 20 of the automobile 2 in Figure 3 includes a vehicle network 30 and a plurality of control devices connected to the vehicle network 30. Figure 4 shows, as examples of the plurality of control devices, a drive control device 21, a steering control device 22, a braking control device 23, a driving control device 24, a driving operation control device 25, a detection control device 26, a communication control device 27, and a call control device 28. In addition to these, the control devices of the automobile 2 may also include, for example, an air conditioning control device, an occupant monitoring control device, a driving position control device, and so on.

[0022] The vehicle network 30 may conform to standards such as CAN (Controller Area Network) or LIN (Local Interconnect Network), which are used in vehicles such as automobile 2. Such a vehicle network 30 has multiple bus cables 32 to which multiple control devices are connected, and a central gateway device 31 which acts as a relay device to which the multiple bus cables 32 are connected. The central gateway device 31 controls the routing between the multiple control devices through the multiple bus cables 32. This allows the control devices to send and receive information with other control devices connected to other bus cables 32.

[0023] The driving control device 25 is connected to the operating components that the driver operates to drive the vehicle 2, such as the steering wheel 41, brake pedal 42, accelerator pedal 43, and shift lever 44. The driving control device 25 detects the driver's operation of the operating components, generates operation information corresponding to the operation, and outputs it to other control devices via the vehicle network 30.

[0024] The driving control device 24 controls the driving of the vehicle 2. For example, the driving control device 24 acquires operation information from the driving operation control device 25 via the vehicle network 30. The driving control device 24 then generates drive information, steering information, and braking information according to the acquired information and outputs them to the drive control device 21, steering control device 22, and braking control device 23 via the vehicle network 30. In this case, the driving control device 24 may generate drive information, steering information, and braking information that directly corresponds to the operation information, or it may generate drive information, steering information, and braking information that has been adjusted to assist the operation. Furthermore, the driving control device 24 may generate drive information, steering information, and braking information for automatic driving that does not require operation by the occupants.

[0025] The drive control device 21 acquires drive information from the vehicle network 30 and controls the engine, motor, transmission, etc. (not shown), which are the power sources of the vehicle 2, to control the magnitude and balance of the driving force of the vehicle 2. The steering control device 22 acquires steering information from the vehicle network 30 and controls the steering motor (not shown) and other components of the vehicle 2 to control the direction of travel. The braking control device 23 acquires braking information from the vehicle network 30 and controls the braking of the vehicle 2 by controlling braking components and motors (not shown) of the vehicle 2. The communication control device 27 communicates with the base station 3 located near the automobile 2 to establish a wireless communication channel. The communication control device 27 uses the wireless communication channel established with the base station 3 to send and receive information with the server device 5 and other devices. The communication control device 28 is connected to a speaker 46 and a microphone 47, which are used by the occupants of the vehicle 2 to communicate with, for example, the operator of the server device 5.

[0026] The detection control device 26 is connected to various sensor components installed in the automobile 2. Figure 4 shows examples of sensor components, including a speed sensor 51, an acceleration sensor 52, an external camera 53, a LiDAR 54, an internal camera 55, and a GNSS receiver 56. The speed sensor 51 detects the speed of the car 2 as it moves. The acceleration sensor 52 detects the acceleration of the car 2 as it moves. The external camera 53 is a camera that captures images of the area outside the vehicle 2. The external camera 53 may be, for example, a 360-degree camera that captures images of the entire area outside the vehicle 2, or it may be a camera that divides the area outside the vehicle 2 into multiple cameras and captures images of that area. Lidar54 is a sensor that uses signals such as infrared to detect the surroundings outside of vehicle 2. Based on reflected infrared waves, Lidar54 generates spatial information about the area outside the vehicle. The GNSS receiver 56 receives radio waves from the GNSS satellite 110 shown in Figure 1 and generates the current position and time of the automobile 2 on which the GNSS receiver 56 is installed. The GNSS satellite 110 may include zenith satellites. The GNSS receiver 56 may also be capable of receiving radio waves from transmitters fixed on the ground.

[0027] The in-car camera 55 is a camera that captures images of the interior of the vehicle 2 where the driver and other occupants are seated. Here, the in-car camera 55 may be a wide-angle camera capable of capturing the driver and passenger in the vehicle 2 in a single image. Such a wide-angle camera that captures multiple occupants in the vehicle 2 should be installed in the central part of the vehicle 2 in the width direction, for example, in the central part of the dashboard.

[0028] The detection control device 26 then outputs the detection information of these sensor components, and the information generated based on the detection information, to other control devices via the vehicle network 30. For example, the detection control device 26 may pre-record information on the facial images of occupants captured by the in-vehicle camera 55 in its memory 64, and then compare this information with the current image captured by the in-vehicle camera 55 to identify occupants in the vehicle 2. In this case, the detection control device 26 can output the information of the identified occupants to other control devices via the vehicle network 30 as information generated based on the detection information. The detection control device 26 may also repeatedly acquire the latest image captured by the in-vehicle camera 55 to monitor the occupants' condition. Occupants may develop illnesses while driving. When the detection control device 26 detects such a change in the occupants' condition, it may send an emergency response request from the communication control device 27 to the server device 5. Such a detection control device 26 functions as an occupant monitoring control device.

[0029] Figure 4 is a basic configuration diagram of the communication control device 27 shown in Figure 3. The other control devices in Figure 3 can have the same basic configuration as those in Figure 4. The communication control device 27 in Figure 4 includes an in-vehicle communication device 61, input / output ports 62, a timer 63, a memory 64, a CPU 65, and an internal bus 66 to which these are connected.

[0030] The in-vehicle communication device 61 is connected to the vehicle network 30. The in-vehicle communication device 61 inputs and outputs information to and from other in-vehicle communication devices 61 of other control devices via the vehicle network 30. The input / output port 62 is connected to various components that are connected to the communication control device 27. In the case of the communication control device 27, a communication device for communication with the base station 3 may be connected to it. Timer 63 measures time or time. The time of Timer 63 may be calibrated by the current time obtained by GNSS receiver 56. This allows the time of the server device 5 and the time of the automobile 2 to match with high accuracy. Memory 64 stores programs executed by the CPU 65 and various other information. Memory 64 may consist of, for example, semiconductor memory, an HDD, etc. Semiconductor memory includes, for example, volatile memory such as RAM, and non-volatile memory such as ROM and SSD. The CPU 65 reads and executes the program stored in memory 64. As a result, the CPU 65 functions as a vehicle control unit, controlling the overall operation of the communication control device 27 in which it is installed.

[0031] Figure 5 is a flowchart showing the control of information transmission for emergency response services using the vehicle control system shown in Figure 3. Here, we will explain assuming that the CPU 65 of the communication control device 27 performs the transmission control shown in Figure 5. The CPU 65 of the communication control device 27 repeatedly performs the transmission control shown in Figure 5, for example, when there are passengers in the automobile 2. Furthermore, the CPUs 65 of various control devices included in the control system 20 of the automobile 2 in Figure 3 may perform the transmission control shown in Figure 5. Alternatively, multiple CPUs 65 of the control system 20 may cooperate to perform the transmission control shown in Figure 5.

[0032] In step ST11, the CPU 65 collects the latest vehicle information of the automobile 2. Here, vehicle information can be any information that can be collected from various parts of the automobile 2. Vehicle information may include information indicating the driving status and malfunctions of the automobile 2, and information indicating the status and malfunctions of the occupants. For example, information indicating the status of the occupants may be images captured by the in-car camera 55. The images captured by the in-car camera 55 may show multiple occupants, such as the driver and passenger, who are riding in the automobile 2.

[0033] In step ST12, the CPU 65 determines whether the vehicle's status, which can be determined based on the vehicle information, requires communication with the server device 5. If there is a malfunction with the vehicle 2 or its occupants, if the vehicle 2 has been involved in an accident, or if the occupants have requested communication through button operation (not shown), the CPU 65 may determine, based on this information, that communication with the server device 5 is required. In this case, the CPU 65 proceeds to step ST13 to perform communication with the server device 5. If the CPU 65 does not determine that communication with the server device 5 is required, it terminates this control.

[0034] From step ST13, the CPU 65 starts processing to perform communication with the server device 5. First, the CPU 65 selects information to transmit from the latest vehicle information it has collected. If there is a malfunction in the vehicle 2, the CPU 65 may select various information about the vehicle 2 as information to transmit. If there is a malfunction in the occupant, or if the occupant is operating a button, the CPU 65 may select various information about the occupant as information to transmit. If an accident has occurred involving the vehicle 2, the CPU 65 may select various information about the vehicle 2 and the occupant as information to transmit. When selecting various information about the occupant, the CPU 65 may select images captured by the in-vehicle camera 55, including the occupant's face.

[0035] In step ST14, the CPU 65 determines whether or not it has selected an image of the occupant's face as the information to be transmitted. If, for example, the CPU 65 has selected an image captured by the in-vehicle camera 55 that includes the occupant's face as the information to be transmitted, it determines that an image of the occupant's face has been selected and proceeds to step ST15. If the image captured by the in-vehicle camera 55 that includes the occupant's face has not been selected as the information to be transmitted, the CPU 65 skips step ST15 and proceeds to step ST16.

[0036] In step ST15, the CPU 65 generates information to be used for face morphing processing based on the crew member's face image, based on the image including the crew member's face that has been selected as the information to be transmitted.

[0037] In step ST16, the CPU 65 transmits the transmission information from the communication control device 27 to the server device 5 via the base station 3. If in step ST15 information is being generated to be used for facial morphing based on the crew member's facial image, the CPU 65 sends the information to be used for morphing instead of the crew member's facial image. In this case, the CPU 65 also transmits information for morphing processing in the server device 5, such as classification information about the occupants and the vehicle. The CPU 65 may acquire information about the occupants and the vehicle from various parts of the control system 20 of the automobile 2, generate classification information about the occupants and the vehicle, and transmit it.

[0038] The classification information about the occupants and vehicles used for morphing processing in server device 5 only needs to be information that does not uniquely identify the occupants. Vehicle classification information that does not identify the occupants includes, for example, the location information of the occupant's vehicle 2, and specification information such as the left / right position of the steering wheel of the occupant's vehicle 2. Furthermore, the classification information for the occupant's vehicle includes, for example, vehicle setting information tailored to the occupant's physique. This vehicle setting information tailored to the occupant's physique includes, for example, information on the occupant's seat position, height, and tilt settings, steering wheel position and height settings, accelerator operation information, and individual occupant settings for vehicle 2. Furthermore, classification information about the crew members themselves includes, for example, whether or not they wear glasses, whether or not they wear a hat, whether or not they have a beard, gender, patterned hairstyle, hair color in a patterned finite division, eye color in a patterned finite division, and skin color in a patterned finite division. None of this various classification information about crew members can identify an individual on its own. Nor can multiple classification pieces of information be combined to uniquely identify a crew member.

[0039] If information used for facial morphing processing based on the crew member's facial image has not been generated in step ST15, the CPU 65 sends the crew member's facial image to the server device 5. In this way, the CPU 65 of the communication control device 27 of the automobile 2 can prevent the images captured by the in-vehicle camera 55, including the occupants' facial images, from being transmitted outside the vehicle. The occupants' facial images may be protected as personal information.

[0040] Figure 6 is a detailed flowchart of the generation control of morphing processing information in step ST15 of Figure 5.

[0041] In step ST61, the CPU 65 extracts a face image, which is the area where the occupant's face is captured, from the image captured by the in-car camera 55, which has already been acquired as vehicle information obtained from various parts of the automobile 2. Here, a crew member's facial image refers to an image that contains the facial image components of a crew member.

[0042] In step ST62, the CPU65 generates point cloud information representing the crew member's face from the extracted crew member's face image. Here, the point cloud information includes multiple points indicating the position of the outline of the crew member's face in the crew member's face image, and multiple points indicating the positions of elements of the crew member's face such as eyebrows, eyes, nose, and mouth. In addition, for example, the point cloud information may also include multiple points indicating the position of the outline of the entire head, including the crew member's head. Furthermore, the multiple points indicating the positions of the facial elements should be in a combination that can indicate the shape and extent of those elements. For example, the multiple points for the eyebrows should be a combination that indicates the positions of both ends of each eyebrow and the positions in between. The multiple points for the eyes should be a combination that indicates the position of the inner corner of the eye, the outer corner of the eye, the upper eyelid, and the lower eyelid. The multiple points for the nose should be a combination that indicates the upper and lower edges of the nose, both left and right edges, and the tip of the nose. The multiple points for the mouth should be a combination that indicates the outer circumference of the upper lip and the outer circumference of the lower lip. Point cloud information of multiple points on the faces of these crew members can be generated through landmark detection processing.

[0043] In step ST63, the CPU 65 generates patch information to divide the face image into multiple parts. In this case, the CPU 65 may use the point cloud information generated in step ST62 as the basis for image division to generate patch information to divide the face image. The CPU 65 may generate multiple patch information for the face image using, for example, the Delaunay method. In the Delaunay method, the face image is divided into multiple parts so that triangular patch regions are obtained.

[0044] In step ST64, the CPU65 generates brightness values ​​for the crew member's face image. The CPU65 may generate average brightness values, representative brightness values, etc., for the face portion of the crew member's face image. Subsequently, in step ST16 of Figure 5, the CPU 65 transmits the information generated in steps ST62 to ST63 to the server device 5 as information to be used for face morphing processing based on the occupant's face image. Instead of the captured image including the occupant's face, the CPU 65 transmits the point cloud information and patch information generated by the processing, along with the classification information of the occupant or vehicle that cannot uniquely identify the occupant as described above, to the server device.

[0045] Figure 7 is a diagram showing the configuration of server device 5 in Figure 1. The server device 5 in Figure 7 includes a server timer 16, server memory 17, server CPU 18, and a server bus 19 to which these are connected. Other devices such as a server GNSS receiver 11, server communication device 12, server display device 13, server operation device 14, and server voice device 15 may also be connected to the server bus 19.

[0046] The server communication device 12 is connected to the communication network 4 of the communication network 6. The server communication device 12 sends and receives information to and from other devices connected to the communication network 4, such as the automobile 2 and the base station 3, via the communication network 4. The server-GNSS receiver 11 receives radio waves from the GNSS satellite 110 shown in Figure 1 to obtain the current time. The server display device 13 outputs information from the server device 5 by display. The server display device 13 can be, for example, an LCD monitor. The server display device 13 can function as an image output unit in the server device 5. The server operation device 14 is a device operated by an operator on the server device 5. The server operation device 14 may be, for example, a keyboard, a touch panel, or the like. The server voice device 15 is a device used by the operator for communication in the server device 5. The server voice device 15 may consist of, for example, a speaker and a microphone. The server timer 16 measures time or time. The time of the server timer 16 may be calibrated by the current time obtained by the server GNSS receiver 11.

[0047] Server memory 17 stores programs executed by the server CPU 18 and various other information. Server memory 17 may consist of, for example, semiconductor memory, an HDD, etc. Semiconductor memory includes, for example, volatile memory such as RAM, and non-volatile memory such as ROM and SSD. Volatile memory stores information temporarily and is suitable for storing, for example, personal information. In this embodiment, the server memory 17 may also record a group of face images 90, which includes a plurality of sample face images, as information for morphing processing in the server device 5. The group of face images 90 includes information on the plurality of sample face images, as well as classification information for each sample face image associated with each sample face image.

[0048] Here, the classification information for the sample face image may consist of classification information for other people in the sample image, corresponding to the classification information for the occupant and the vehicle 2 transmitted from the vehicle 2. The classification information for other people in such sample facial images could include, for example, information about the region to which the other person belongs, and information about the specifications of the car in that region. Furthermore, classification information about other people's vehicles includes, for example, vehicle settings information tailored to the other person's physique. This vehicle settings information tailored to the other person's physique includes, for example, information on the other person's seat position, height, and tilt settings, steering wheel position and height settings, accelerator operation information, and personalized settings information for vehicle 2. Furthermore, classification information about the other person themselves includes, for example, whether or not they wear glasses, whether or not they wear a hat, whether or not they have a beard, gender, patterned hairstyle, hair color in a patterned number of finite divisions, eye color in a patterned number of finite divisions, skin color in a patterned number of finite divisions, and so on. These various classifications of others are not information that can uniquely identify an individual on their own. Furthermore, even combining multiple classifications does not result in information that can uniquely identify an individual. The classification information of other users for the sample images may include all or some of the various types of information described above.

[0049] The server CPU 18 reads and executes the program stored in the server memory 17. This allows the server CPU 18 to function as a server control unit, controlling the overall operation of the server device 5. The server CPU 18, acting as the server control unit, may, for example, manage and control the temporary recording of personal information in the server memory 17. For instance, the server CPU 18 may delete personal information from the server memory 17 once the provision of vehicle services has ended. Furthermore, the server CPU 18 may manage and control the sending and receiving of information with the vehicle 2 using the server communication device 12. For example, when the server communication device 12 receives information from the vehicle 2, the server CPU 18 may execute control according to the received information, such as control for vehicle services. The server CPU 18 may output a face image based on the occupant's face to the server display device 13, which acts as an image output unit. As a result, the server device 5 can receive information from the vehicle 2 and provide vehicle services based on the received information.

[0050] Figure 8 is a flowchart showing the control of the generation of processed images for emergency response services by the server CPU 18 of the server device 5 shown in Figure 7. The server CPU 18 of server device 5, acting as the server control unit, repeatedly performs the control of generating the processed image shown in Figure 8.

[0051] In step ST71, the server CPU 18 acquires morphing processing information. In step ST16 of Figure 5, the CPU 65 of the vehicle 2 transmits information to the server device 5 that will be used for facial morphing processing based on the occupant's facial image. The server communication device 12 of the server device 5 receives this information used for morphing processing. The information used for morphing processing includes point cloud information and patch information of the occupant's face, as well as classification information of the occupant or vehicle that does not uniquely identify the occupant. The server CPU 18 may acquire this occupant or vehicle classification information as morphing processing information.

[0052] From step ST72, the server CPU 18 starts the process of selecting sample face images to be used for morphing.

[0053] In step ST73, the server CPU 18 sets the conditions for acquiring a sample face image from the group of face images 90 recorded in the server memory 17. The server CPU 18 may set the occupant or vehicle classification information acquired from the automobile 2 in step ST71 as a condition for acquiring a sample face image. The server CPU 18 may set the acquisition conditions for sample face images as, for example, the location information of the occupant's vehicle 2 included in the occupant or vehicle classification information, specification information such as the left / right position of the steering wheel of the occupant's vehicle 2, and vehicle setting information according to the occupant's physique. Furthermore, the server CPU 18 may set the classification information about the occupant itself, which is included in the occupant or vehicle classification information, as a condition for acquiring sample facial images.

[0054] In step ST74, the server CPU 18 acquires a sample face image from the face image group 90 in the server memory 17 under the set acquisition conditions. This allows the server CPU 18 to select, for example, a sample face image from among multiple sample face images recorded in the server memory 17 that is associated with information corresponding to the location information or vehicle specification information of the receiving occupant.

[0055] In step ST75, the server CPU 18 determines whether to terminate the process of acquiring sample face images from the face image group 90 in the server memory 17. The server CPU 18 may decide to terminate the sample face image acquisition process if it has finished processing all of the multiple sample face images included in the face image group 90 in the server memory 17. Alternatively, the server CPU 18 may decide to terminate the sample face image acquisition process if it has finished selecting a predetermined number of sample face images from the face image group 90 in the server memory 17. If the server CPU 18 does not decide to terminate the sample face image acquisition process, it returns to step ST73. The process from step ST73 to step ST75 is repeated, and if it is decided to terminate the sample face image acquisition process, the server CPU 18 proceeds to step ST76. In this case, the server CPU 18 basically selects multiple sample face images from the face image group 90 in the server memory 17 that are associated with information corresponding to the received occupant or vehicle classification information.

[0056] In step ST76, the server CPU 18 determines whether it has acquired multiple sample face images from the face image group 90 in the server memory 17. If multiple sample face images have been selected, the server CPU 18 proceeds to step ST77. If only one sample face image has been selected, the server CPU 18 finishes the process of selecting the sample face image to be used for morphing and proceeds to step ST79.

[0057] In step ST77, the server CPU 18 determines the similarity between multiple sample face images obtained from the face image group 90 in the server memory 17 and the face images of the crew members. The server CPU 18 may determine the similarity between multiple sample face images obtained and the face images of the crew members by comparing the brightness values ​​of the acquired sample face images with the brightness values ​​of the crew members included in the crew member classification information.

[0058] In step ST78, the server CPU 18 selects one sample face image to be used for morphing from multiple sample face images obtained from the face image group 90 in the server memory 17. The server CPU 18 may select the sample face image to be used for morphing that it determined to have the highest similarity to the crew member's face image in the process of step ST77. With this, the server CPU 18 finishes the process of selecting a sample face image to be used for morphing. If there are multiple sample face images to be selected using classification information of a occupant or vehicle that does not allow for the unique identification of an occupant, the server CPU 18 can further select one sample image to be used for morphing processing based on the similarity of the brightness values ​​of the face images.

[0059] In step ST79, the server CPU 18 performs morphing on the sample face image. The server CPU 18 generates a morphed face image by performing morphing on the point cloud information and patch information of the selected sample face image and the received point cloud information and patch information of the crew.

[0060] In step ST80, the server CPU 18 outputs the morphed face image generated by the morphing process to the server display device 13.

[0061] Next, we will explain the series of processes described above using specific examples of crew member facial images and sample facial images.

[0062] Figure 9 is an explanatory diagram illustrating an example of an image 70 captured by the in-vehicle camera 55 shown in Figure 4. Since the in-car camera 55 in Figure 4 is a wide-angle camera, the image 70 captured by the in-car camera 55 in Figure 9 captures multiple occupants, including the driver and passenger, who are in the car 2. Even if each occupant moves around inside the car, their entire head is captured in the image 70. In this case, in step ST61 of Figure 7, the CPU 65 extracts the image area including the driver's face, for example, shown by the dashed frame in the figure, as the driver's face image 71.

[0063] Figure 10 is an explanatory diagram of an example of a face image 71, which is the imaged region of the driver's face extracted from the captured image 70 in Figure 9. In the driver's face image 71 in Figure 10, the entire head of the driver is included without being cut off at the outer edge of the image. The CPU 65 may also crop the driver's face image 71 to include only the driver's eyebrows to chin, rather than the entire driver's head. Even in this case, the driver's face image 71 still contains the image components of the driver's face. The CPU 65 can then generate point cloud information for the elements of the driver's face from the eyebrows to the chin, as will be described later. However, even when the head portion is cut off in this way, the CPU 65 crops the driver's face image 71 so that a predetermined margin 77 is included on the left and right sides of the driver's face area. The margin 77 is useful in the processing described later.

[0064] Figure 11 is an explanatory diagram illustrating an example of point cloud information generated for the driver's face image 71 in Figure 10. The CPU 65 can generate point cloud information as shown in Figure 11 for the driver's face image 71 in Figure 10 in step ST62 in Figure 7. The point cloud information in Figure 11 includes multiple points 72 representing the face itself, such as multiple points indicating the position of the driver's facial contour and multiple points indicating the position of elements of the driver's face such as eyebrows, eyes, nose, and mouth, as well as multiple points 73 representing the outer edge, which is the contour of the face image itself. Here, multiple points 73 on the outer edge of the image are provided at each corner and at the center of each side of the outer edge of the rectangular image. Furthermore, the multiple points 73 on the outer edge of the image are separated from the multiple points 72 on the driver's face itself because the margins 77 on the left and right of the driver's face image 71 are included. Furthermore, multiple points 73 on the outer edge of the image are positioned to surround multiple points 72 on the driver's face itself from the outside.

[0065] Figure 12 is an explanatory diagram illustrating an example of patch information generated for the driver's face image 71 in Figure 10. In the patch information shown in Figure 12, each patch is represented as a triangle. The CPU 65 can generate the patch information shown in Figure 12 for the driver's face image 71 in Figure 10 in step ST63 in Figure 7, using the Delaunay method. The patch information in Figure 12 includes a patch 84 representing the driver's face itself, as well as a patch 85 representing the area outside the driver's face. The CPU 65 can divide the entire face image into multiple triangular patches, as shown in Figure 12, by processing the Delaunay method with multiple points 72 representing the driver's face itself, as well as multiple points 73 located on the outer edge of the image. Because there are margins 77 on the left and right of the driver's face image 71, the CPU 65 can divide the entire face image into multiple triangular patches without affecting the division of the face itself into multiple patches. In contrast, if, for example, there are no margins 77 on the left and right sides of the driver's face image 71, the CPU 65 is more likely to perform a different division than shown in Figure 12 when dividing the face image into multiple patches using the Delaunay method. For example, there is a higher possibility of performing a patch division that is undesirable as face patch information, such as treating the points on the left and right outer edges of the image as part of the face contour. To avoid such a situation, in this embodiment, as shown in Figure 10 and Figure 13 described later, the face image is cropped so that the width of the face is about 1 / 3 of the width of the face image. For example, by providing margins on both the left and right sides of the face that are at least 25% or more of the width of the face image, the CPU 65 can generate desirable face patch information as shown in Figure 12 using the Delaunay method. The CPU 65 should crop an imaging area that is larger than the driver's face in at least the width direction of the driver's face as the imaging area for the driver's face.

[0066] In this way, during the cropping process, the CPU 65 crops an image area from the captured image 70 acquired from the in-vehicle camera 55 that is larger than the occupant's face in at least the width direction of the occupant's face, and generates a face image as the image area of ​​the occupant's face. Then, during the point cloud generation process, the CPU 65 generates multiple point cloud pieces for the outer edge of the cropped image area of ​​the occupant's face, including positions away from the occupant's face. Furthermore, during the patch generation process, the CPU 65 uses the multiple point cloud pieces at the outer edge of the cropped image area of ​​the occupant's face, along with the face point cloud information indicating the contour of the face and the positions of the facial elements, as a basis for image division, and generates patch information that divides the cropped image area of ​​the occupant's face into units based on triangles. The CPU 65 generates this information used for occupant morphing and transmits it to the server device 5 along with occupant or vehicle classification information that does not uniquely identify the occupant.

[0067] Next, using Figures 13 to 15, we will explain the information of the sample face image 91 that the CPU 65 acquires in step ST65 of Figure 17.

[0068] Figure 13 is an explanatory diagram illustrating an example of a combination of multiple sample facial images 91-94. Multiple sample face images 91-94 may be recorded in the server memory 17 as a face image group 90. Then, the group of face images 90 recorded in the server memory 17 contains multiple face images of other people, which are recorded as sample face images 91-94. Figure 13 specifically shows examples of sample faces: 91 of a first person with fair skin and a round face, 92 of a second person with dark skin and a long face, 93 of a third person with dark skin and a round face, and 94 of a fourth person with dark skin and a long face wearing glasses.

[0069] Furthermore, each sample face image 91-94 contains classification information 95-98 about the other person included in that image. This classification information of others, 95-98, includes the classification information of others for the sample image, corresponding to the classification information of the occupants and vehicle 2 transmitted from vehicle 2. For example, classification information 95 for the first other with fair skin and a round face includes that they are a resident of Japan, have a small build and sit further forward in the seat, have a yellowish brightness value for their face, and do not wear glasses. The automobile 2 sold in Japan is right-hand drive. The classification information 96 for the sample face image 92 of a second person with a dark complexion and long face includes that the person is a resident of the United States, has a large build and sits further back in the seat, has a brownish luminance value for their face, and does not wear glasses. The car 2 sold in the United States is left-hand drive. The classification information 97 for the sample face image 93 of a third party with dark skin and a round face includes that the person is a resident of the United States, has a small build and is positioned forward of the seat, has a brownish luminance value for their face, and is not wearing glasses. The classification information 97 for the sample face image 94 of the fourth other, who is dark-skinned, has a long face, and wears glasses, includes the following: being a resident of the United States, being large in build and positioned further back from the seat, having a brownish luminance value for the face, and wearing glasses.

[0070] While the information of this group of facial images 90 is recorded in the server memory 17, the server CPU 18 receives and acquires occupant or vehicle classification information as morphing processing information from the automobile 2 in step ST71. Furthermore, the classification information C1 for the occupant or vehicle, as shown in the diagram, includes location information in Japan for the occupant's vehicle 2, and specification information for vehicle 2 destined for Japan, including information on right-hand drive. In this case, the server CPU 18 can use the location information and specification information of the occupant's car 2 to select a sample face image 91 of at least one other person with a fair complexion and round face from the information of the face image group 90 in Figure 13. In this way, the server CPU 18 can select from among multiple sample face images 91 to 94 recorded in the server memory 17 a sample face image 91 that is associated with information corresponding to the location information or specification information of the occupant's vehicle 2 that is being received.

[0071] Furthermore, the classification information C2 for occupants or vehicles shown in the figure includes, in addition to the information in classification information C1, information on the seat setting position according to the occupant's physique, specifically information indicating a forward position. In this case, the server CPU 18 can use the seat setting position in addition to, or instead of, the location and specification information of the occupant's car 2, to select a sample face image 91 of at least a fair-skinned, round face from the information of the face image group 90 in Figure 13. In this way, the server CPU 18 can select from among multiple sample face images 91 to 94 recorded in the server memory 17 a sample face image 91 that is associated with the setting information of the vehicle 2 corresponding to the physique of the occupant being received.

[0072] Furthermore, the classification information C3 for the occupant or vehicle, as shown in the figure, includes information about the absence of glasses on the occupant, in addition to the information in classification information C2. In this case, the server CPU 18 can use information about the location, specifications, and seat position of the occupant's car 2, as well as information about whether or not the occupant is wearing glasses, to select a sample face image 91 of at least a fair-skinned, round face from the information of the face image group 90 in Figure 13. In this way, the server CPU 18 can select from among multiple sample face images 91 to 94 recorded in the server memory 17 a sample face image 91 that is associated with the classification information of the crew member it is receiving.

[0073] Figure 14 is an explanatory diagram illustrating an example of point cloud information generated for the sample face image 91 selected in Figure 13. Figure 15 is an explanatory diagram illustrating an example of patch information generated for the sample face image 91 selected in Figure 13.

[0074] In the first sample face image 91 of another person in Figure 13, the entire head of the other person is contained within the image without being cut off at the outer edge, with a margin around the head. In this case, the server CPU 18 can generate point cloud information for the first sample face image 91 of the first other person, as shown in Figure 14, by performing a point cloud generation process on the first sample face image 91 of the first other person. The point cloud information for the first sample face image 91 in this case includes multiple points 82 representing the face itself, indicating the contours of the face and the positions of the facial elements, as well as multiple points 83 located on the outer edge of the face image. Furthermore, the server CPU 18 generates patch information for the first other person's sample face image 91 through a patch generation process, as shown in Figure 15, which divides the first other person's sample face image 91 into units based on triangles. The patch information includes a patch 84 that divides the other person's face itself, as well as a patch 85 that divides the area outside the other person's face. The server CPU 18 may then generate the point cloud information shown in Figure 14 and the patch information shown in Figure 15 for the first sample face image 91 of another person shown in Figure 13, by processing similarly to the processing for the driver face image 71 shown in Figure 10.

[0075] Figure 16 is an explanatory diagram of an example of a morphing face image 100 that is displayed and output in place of the crew member's face image. This morphing face image 100 is an image that is displayed and output by the server device 5 in place of the driver's face image 71. In step ST79 of Figure 8, the server CPU 18 performs morphing on the first sample face image 91 of another person. In the morphing process, the server CPU 18 uses the information from Figures 13 to 15 regarding the sample face image 91 and the information from Figures 10 to 12 regarding the driver's face image 71. Multiple points 72, 73 included in the point cloud information of the driver's face image 71 in Figure 11 correspond to multiple points 82, 83 included in the point cloud information of the sample face image 91 in Figure 14. Furthermore, the multiple patches 74 and 75 included in the patch information of the driver's face image 71 in Figure 12 correspond to the multiple patches 84 and 85 included in the patch information of the sample face image 91 in Figure 15. Thus, the point cloud information and patch information of the sample face image 91 are generated to correspond well with the point cloud information and patch information of the driver's face image 71.

[0076] In this case, the server CPU 18, for example, moves the positions of each point 82, 83 in the point cloud information of the sample face image 91 closer to the corresponding positions of points 72, 73 in the point cloud information of the driver's face image 71. As a result, the position and range of each patch 84 and 85 in the patch information of the sample face image 91 will change so that they overlap with the position and range of each patch 74 and 75 in the patch information of the driver's face image 71. As a result, the face of the first other person in the sample face image 91 of the first other person in Figure 13 approaches the face of the driver in image 71 of Figure 10.

[0077] In this way, the server CPU 18 performs morphing on the sample face image 91 so that the point cloud information and patch information of the sample face image 91 become closer to the point cloud information and patch information of the driver's face image 71. Here, if the positions of points 82 and 83 in the point cloud information of the sample face image 91 coincide with the positions of points 72 and 73 in the point cloud information of the driver's face image 71, the first other person's sample face image 91 will be morphed to a 100% degree. The facial contour, position, and size of the facial elements in the sample face image 91 may be approximately the same as those in the actual driver's face image 71. However, since the sample face image 91 used for the morphing process is not that of the driver himself but of a different person, even if the morphing process is performed to a 100% degree, the morphed face image 100 will not be the driver's face image 71 itself. If the morphing rate is 50%, the morphed face image 100 will be an intermediate between the driver's face in image 71 of Figure 10 and the other person's face in sample image 91 of Figure 13, as shown in Figure 16. Even in this case, the morphed face image 100 contains the driver's facial features and expressions in proportion to the morphing. The server CPU 18 performs morphing on the sample face image 91, setting the morphing ratio to an arbitrary value. The morphing ratio value may be fixed or it may be arbitrarily set by a driver or other means.

[0078] As described above, in the vehicle service provision system 1 of this embodiment, the CPU 65 of the occupant's vehicle 2 and the server CPU 18 of the server device 5 cooperate to perform occupant image acquisition processing, cropping processing, point cloud generation processing, patch generation processing, sample face image acquisition processing, and morphing processing. The server CPU 18 then outputs the morphed face image produced by the morphing processing from the output unit of the server device 5 in place of the occupant's face image. As a result, the output unit of the server device 5 can output a face image that reflects the occupant's state, such as their facial expression. Consequently, the operator can grasp the occupant's current actual state and facial expression based on the output face image. In contrast, if, for example, the crew's facial images are abstracted, replaced, or masked, it becomes difficult for the operator to accurately grasp the crew's current actual state and facial expressions from the facial images output by the server device 5. Furthermore, the CPU 65 of the occupant's vehicle 2 performs occupant image acquisition processing, cropping processing, point cloud generation processing, and patch generation processing. The CPU 65 of the occupant's vehicle 2 then transmits the information generated by these processes to the server device 5 in place of the captured image including the occupant's face. As a result, the vehicle 2 of this embodiment can avoid transmitting the occupant's face image itself, which needs to be protected as personal information of the occupant, to the outside. Thus, in this embodiment, it is possible to protect personal information while minimizing any impairment to the convenience of the vehicle service provision system 1.

[0079] Furthermore, in this embodiment, the server device 5 includes a server memory 17 and a server CPU 18, along with an image output unit. The server memory 17 records information on multiple sample face images 91 to 94, which include point cloud information 82, 83 and patch information 84, 85 as information on face images different from the occupant extracted from the captured image, and which are associated with classification information 95 to 98 of the sample face images. The server CPU 18 then selects one or more sample face images from the multiple sample face images 91 to 94 recorded in the server memory 17, using occupant or vehicle classification information C1 to C3, which is received from the CPU 65 of the occupant's vehicle 2 and does not uniquely identify the occupant, and which are associated with information corresponding to the occupant or vehicle classification information C1 to C3. In such a server device 5, it is possible to record a large number of sample facial images in the server memory 17 that can be used effectively for each crew member. Moreover, even if a large number of sample face images are recorded in the server memory 17, the server CPU 18 can quickly narrow down and select the sample face images to be used for morphing processing through a simple and low-load process called selection processing based on the classification information C1 to C3 of the occupants or vehicles. The server CPU 18 can quickly select sample face images that can correspond well to the occupants for morphing processing through a less loady process than performing a high-load process such as comparing and selecting based on the similarity of the brightness values ​​of the faces for all of the numerous sample face images recorded in the server memory 17. If other people in the sample face images correspond well to the occupants, the resulting morphed face image is likely to be closer to the occupants than if they did not correspond well.

[0080] The embodiments described above are examples of preferred embodiments of the present invention, but the present invention is not limited thereto, and various modifications or changes are possible without departing from the spirit of the invention. [Explanation of Symbols]

[0081] 1...Vehicle service provision system, 2...Automobile (vehicle), 3...Base station, 4...Communication network, 5...Server equipment, 6...Communication network, 9...Roadside service, 11...Server GNSS receiver, 12...Server communication device, 13...Server display device, 14...Server operation device, 15...Server voice device, 16...Server timer, 17...Server memory, 18...Server CPU, 19...Server bus, 20...Control system, 21...Drive control device, 22...Steering control device, 23...Brake control device, 24...Driving control device, 25...Driving operation control device, 26...Detection control device, 27...Communication control device, 28...Call control device, 30...Vehicle network, 31...Central gateway device, 32...Bus cable 41...Steering wheel, 42...Brake pedal, 43...Accelerator pedal, 44...Shift lever, 46...Speaker, 47...Microphone, 51...Speed ​​sensor, 52...Accelerometer, 53...Exterior camera, 54...Lidar, 55...Interior camera, 56...GNSS receiver, 61...In-vehicle communication device, 62...Input / output port, 63...Timer, 64...Memory, 65...CPU, 66...Internal bus, 70...Captured image, 71...Driver's face image, 72,73,82,83...Points, 74,75,84,85...Patches, 77...Margin area, 90...Face image group, 91~94...Sample face image, 95~98...Classification information, 100...Morphing face image, 110...GNSS satellite, C1~C3...Classification information of occupants or vehicle

Claims

1. A system that transmits information from a vehicle to a server device, and provides services based on the information received by the server device, The vehicle has an in-vehicle camera and a vehicle control unit capable of capturing images of the vehicle's occupants, and the server device has an image output unit, a server memory and a server control unit. The vehicle control unit, The process involves acquiring an image of the occupants, including their faces, from the in-vehicle camera, and From the acquired image, an extraction process is performed to cut out the image region of the crew member's face, A point cloud generation process that generates point cloud information of the face, indicating the contour of the face and the position of facial elements within the imaged area of ​​the extracted crew member's face, The generated point cloud information is used as the basis for image segmentation, and a patch generation process is performed to generate patch information that divides the imaged area of ​​the extracted crew member's face. The process is executed to send, in place of the captured image including the occupant's face, the point cloud information and patch information generated by the process, along with classification information of the occupant or the vehicle that cannot uniquely identify the occupant, to the server device. The aforementioned server memory is The system records information on multiple sample face images, which include point cloud information and patch information as information on face images different from the crew member extracted from the aforementioned captured image, and which are associated with the classification information of the sample face image. The server control unit, A sample image acquisition process selects a sample face image from among a plurality of sample face images recorded in the server memory, using the classification information of the occupant or the vehicle received from the vehicle control unit, to which information corresponding to the classification information of the occupant or the vehicle is associated. A morphing process is performed using the point cloud information and patch information for the selected sample face image and the point cloud information and patch information generated for the occupant to generate a morphed face image. Execute the above morphing process to obtain a morphed face image It is possible to output this from the output unit in place of the facial image of the vehicle's occupant. Vehicle service provision system.

2. In the server memory, the sample face image is recorded as classification information for the sample face image, associated with at least one vehicle classification information from among the information of the region to which the other person in the sample face image belongs, and the vehicle specification information in the region. The vehicle control unit, As classification information for a occupant or vehicle that cannot uniquely identify the occupant, at least one of the occupant's vehicle location information and the vehicle's specification information is transmitted to the server device. The server control unit, in the sample image acquisition process, From among a plurality of sample facial images recorded in the server memory, a sample facial image is selected that is associated with information corresponding to the location information or specification information of the vehicle of the receiving occupant. A vehicle service provision system according to claim 1.

3. In the server memory, the sample face image is further associated with and recorded vehicle setting information corresponding to the physique of the other person in each sample face image, as classification information for the sample face image. The vehicle control unit, As classification information for occupants or vehicles that cannot uniquely identify the occupants, the vehicle setting information corresponding to the occupants' physiques is transmitted to the server device. The server control unit, in the sample image acquisition process, From among multiple sample facial images recorded in the server memory, a sample facial image is selected that is associated with the vehicle setting information corresponding to the physique of the receiving occupant. A vehicle service provision system according to claim 2.

4. In the server memory, the sample face image is further associated with and classified as classification information for the sample face image, which is information that does not uniquely identify other people in each sample face image, and classification information for other people in the sample face image is recorded in a manner that allows for classification. The vehicle control unit, As classification information for crew members or vehicles that cannot be uniquely identified, the crew classification information is transmitted to the server device. The server control unit, in the sample image acquisition process, From among the multiple sample facial images recorded in the server memory, a sample facial image is selected that is associated with the information corresponding to the received crew classification information. A vehicle service provision system according to claim 3.

5. The vehicle control unit further generates brightness values ​​for the occupant's facial image and transmits them to the server device along with the occupant's classification information. The server control unit, in the sample image acquisition process, If there are multiple sample face images to be selected using classification information for a occupant or vehicle that cannot uniquely identify an occupant, the luminance values ​​of the multiple acquired sample face images are compared with the luminance values ​​of the occupants included in the occupant classification information to determine the similarity to the occupant's face image and select one sample image to be used for morphing. A vehicle service provision system according to claim 4.

Images