Image processing device, image processing method, image processing system, and program

Abstract

An image processing device is provided with: an image attribute acquisition unit that acquires an image attribute, which is a manner of capturing an input image; an image conversion unit that performs anonymization processing on the input image; and an image determination unit that determines whether or not the input image on which the anonymization processing has been performed satisfies a predetermined requirement, the image determination unit performing a predetermined process on the input image on which the anonymization processing has been performed when it is determined that the input image on which the anonymization processing has been performed satisfies the predetermined requirement, and performing a predetermined process on the input image on which the anonymization processing has been performed when it is determined that the input image on which the anonymization processing has been performed satisfies the predetermined requirement. The predetermined requirements are determined according to the acquired image attributes.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method, an image processing system and a program. Background Art

[0002] In recent years, efforts to provide a sustainable transportation system that also takes into account people who are particularly vulnerable among traffic participants have been active. To achieve this, research and development efforts are being made to further improve the safety and convenience of traffic through research and development related to autonomous driving technology. For example, in the past, it was known that a technology was used to annotate an individual's facial image in order to generate learning data used in learning a machine learning model. Patent document 1 discloses a technology that generates a synthetic facial image by referring to facial images of multiple people stored in a facial image database, and an annotation operation can be performed on the generated synthetic facial image.

[0003] Prior Art Literature

[0004] Patent Literature

[0005] Patent Document 1: Patent No. 5930450 Summary of the invention

[0006] Problems to be solved by the invention

[0007] The technology described in Patent Document 1 protects the privacy of multiple people by annotating a synthetic facial image synthesized from facial images of multiple people. However, in the prior art, since the original image is transformed to protect privacy, feature information of the original image is sometimes missing. As a result, it is sometimes impossible to generate learning data effective for learning a machine learning model while protecting the privacy of the person reflected in the facial image.

[0008] The present invention was completed in consideration of such a situation, and one of its purposes is to provide an image processing device, an image processing method, an image processing system, and a program that can generate learning data effective for learning a machine learning model while protecting the privacy of a person reflected in a facial image, thereby contributing to the development of a sustainable transportation system.

[0009] Solutions to Solve Problems

[0010] The image processing device, image processing method, image processing system, and program according to the present invention employ the following configurations.

[0011] (1): An image processing device according to one aspect of the present invention comprises: an image attribute acquisition unit for acquiring an image attribute, wherein the image attribute is a shooting method of an input image; an image conversion unit for performing anonymization processing on the input image; and an image determination unit for determining whether the input image subjected to the anonymization processing satisfies a prescribed requirement, wherein the image determination unit performs a prescribed processing on the input image subjected to the anonymization processing if it is determined that the input image subjected to the anonymization processing satisfies the prescribed requirement, wherein the prescribed requirement is determined based on the acquired image attribute.

[0012] (2): In the above-mentioned aspect (1), the predetermined process is a process of storing the input image subjected to the anonymization process as a target image for annotation work.

[0013] (3): In the above-mentioned aspect (1), the predetermined processing is processing of storing the input image subjected to the anonymization processing as learning information for generating a behavior prediction model, wherein the behavior prediction model is a model for predicting the behavior of a person reflected in the input image.

[0014] (4): In the above-mentioned aspect (1), the predetermined process is a process of transmitting the input image subjected to the anonymization process to an image server via a communication means.

[0015] (5): In the above-mentioned aspect (1), the image attribute is information indicating at least whether the input image is an image captured from inside a vehicle or an image captured from outside the vehicle, and the vehicle is equipped with a camera that captured the input image.

[0016] (6): In the scheme of (5) above, the anonymization processing includes changing the face of a person reflected in the input image to the face of another person, and the prescribed requirements include whether the line of sight direction of the face of the person is consistent with the line of sight direction of the face of the other person when the image attribute indicates that the image is taken of the interior of the vehicle.

[0017] (7): In the scheme of (6) above, the prescribed requirements include whether the line of sight direction of the face of the person is consistent with the line of sight direction of the face of the other person, and whether the facial direction of the face of the person is consistent with the facial direction of the face of the other person, when the image attributes indicate that the image is taken of the interior of the vehicle. The prescribed requirements include whether the facial direction of the face of the person is consistent with the facial direction of the face of the other person, when the image attributes indicate that the image is taken of the exterior of the vehicle.

[0018] (8): In the above aspect (6), when the image attribute indicates that the image is an image of the exterior of the vehicle, the prescribed requirement does not include whether the gaze direction of the face of the person matches the gaze direction of the face of the other person.

[0019] (9): In the above-mentioned aspect (1), when the image determination unit determines that the input image subjected to the anonymization process does not satisfy the predetermined requirement, the image conversion unit performs the anonymization process on the input image again.

[0020] (10): In the above-mentioned aspect (1), when the image determination unit determines that the input image subjected to the anonymization process does not satisfy the predetermined requirement, the image conversion unit does not perform the predetermined process on the input image subjected to the anonymization process.

[0021] (11): Another embodiment of the present invention relates to an image processing system comprising: an image attribute acquisition unit for acquiring an image attribute, wherein the image attribute is a shooting method of an input image; an image conversion unit for performing anonymization processing on the input image; and an image determination unit for determining whether the input image subjected to the anonymization processing satisfies a prescribed requirement, wherein the image determination unit performs a prescribed processing on the input image subjected to the anonymization processing if it is determined that the input image subjected to the anonymization processing satisfies the prescribed requirement, wherein the prescribed requirement is determined based on the acquired image attribute.

[0022] (12): Another embodiment of the present invention relates to an image processing method that causes a computer to perform the following processing: obtaining an image attribute, wherein the image attribute is a shooting method of an input image; performing anonymization processing on the input image; determining whether the input image subjected to the anonymization processing satisfies a prescribed requirement; and, if it is determined that the input image subjected to the anonymization processing satisfies the prescribed requirement, performing prescribed processing on the input image subjected to the anonymization processing, wherein the prescribed requirement is determined based on the obtained image attribute.

[0023] (13): A program according to another embodiment of the present invention causes a computer to perform the following processing: obtaining an image attribute, wherein the image attribute is a shooting method of an input image; performing anonymization processing on the input image; determining whether the input image subjected to the anonymization processing satisfies a prescribed requirement; and, if it is determined that the input image subjected to the anonymization processing satisfies the prescribed requirement, performing prescribed processing on the input image subjected to the anonymization processing, wherein the prescribed requirement is determined based on the obtained image attribute.

[0024] Effects of the Invention

[0025] According to (1) to (13), it is possible to generate learning data effective for learning a machine learning model while protecting the privacy of a person reflected in a facial image. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 1 is a diagram showing an overview of a system 1 including an image processing apparatus 100 according to the present embodiment.

[0027] Figure 2 1 is a diagram showing an example of the functional configuration of the image processing device 100 according to the present embodiment.

[0028] Figure 3 The diagram shows an example of an in-vehicle image and an out-vehicle image acquired from the vehicle M1.

[0029] Figure 4 It is a diagram for explaining the processing executed by the image processing unit 130 .

[0030] Figure 5 It is a diagram for explaining the processing executed by the image conversion unit 140 .

[0031] Figure 6 FIG. 1 is a diagram showing an example of a time series of vehicle interior images converted by the image conversion unit 140 .

[0032] Figure 7 It is a diagram for explaining the processing executed by the image determination unit 150 .

[0033] Figure 8 This is a diagram showing an example of annotation work performed by an annotator.

[0034] Fig. 9 1 is a diagram showing an example of driving assistance using the learned model 180 .

[0035] Fig.10 It is a diagram showing an example of the flow of processing executed by the image conversion unit 140 .

[0036] Fig.11 It is a diagram showing an example of the flow of processing executed by the image determination unit 150 . DETAILED DESCRIPTION

[0037] Hereinafter, embodiments of an image processing device, an image processing method, an image processing system, and a program according to the present invention will be described with reference to the accompanying drawings.

[0038] [summary]

[0039] Figure 1 1 is a diagram showing an overview of a system 1 including an image processing device 100 according to the present embodiment. Figure 1 As shown, the system 1 includes at least one vehicle M1 and vehicle M2, an image processing device 100, and a terminal device 200. For convenience of explanation, the vehicle M1 and vehicle M2 are shown as different vehicles, but these vehicles may be the same vehicle.

[0040] The vehicle M1 is a four-wheel drive vehicle such as a hybrid vehicle or an electric vehicle, and includes at least a camera for photographing the interior of the vehicle M1 and a camera for photographing the exterior of the vehicle M1. While driving, the vehicle M1 transmits the interior and exterior images photographed by these cameras to the image processing device 100 via a network NW such as a cellular network, a Wi-Fi network, and the Internet.

[0041] The image processing device 100 is a server device that, when receiving captured image data including an in-vehicle image and an out-vehicle image from the vehicle M1, performs image conversion described below on the received captured image data. The image conversion is a process for protecting the privacy of people reflected in the in-vehicle image and the out-vehicle image. The image processing device 100 transmits the obtained converted image data to the terminal device 200 via the network NW.

[0042] The terminal device 200 is a terminal device such as a desktop computer or a smart phone. When the user of the terminal device 200 obtains the transformed image data from the image processing device 100, the user performs an annotation assignment operation to the obtained transformed image data, which will be described later. When the annotation assignment operation is completed, the user of the terminal device 200 transmits the annotated image data in which the annotation is assigned to the transformed image data to the image processing device 100.

[0043] When the image processing device 100 receives the annotated image data from the terminal device 200, the received annotated image data is used as learning data, and an arbitrary machine learning model is used to generate a learned model described later. The learned model is, for example, a model that outputs a predicted action (trajectory) of a person shown in the image outside the vehicle for input of an image outside the vehicle, or a model that urges a pedestrian shown in the image outside the vehicle to pay attention to the input of an image inside the vehicle and an image outside the vehicle, taking into account the line of sight of the driver shown in the image inside the vehicle.

[0044] It should be noted that the image data used as learning data at this time may be annotated image data in which annotations are given to transformed image data, or may be annotated image data in which the transformed image data is transformed into captured image data while the annotations are kept unchanged (i.e., annotated image data in which annotations are given to captured image data). By using annotated image data in which annotations are given to captured image data as learning data, it is possible to use learning data that is more in line with reality and in which the influence caused by image transformation is eliminated.

[0045] When the image processing device 100 generates a learned model, it distributes the generated learned model to the vehicle M2 via the network NW. Similar to the vehicle M1, the vehicle M2 is a four-wheel drive vehicle such as a hybrid vehicle or an electric vehicle. When the vehicle M2 is driving, the learned model is input into at least one of the in-vehicle image and the out-vehicle image captured by the camera, thereby obtaining the behavior prediction data of the people around the vehicle M2. The driver of the vehicle M2 can refer to the obtained behavior prediction data and use it for driving the vehicle M2. The following describes the details of each process.

[0046] [Functional Structure of Image Processing Device]

[0047] Figure 2 1 is a diagram showing an example of the functional structure of the image processing device 100 involved in the present embodiment. The image processing device 100, for example, includes a communication unit 110, a transceiver control unit 120, an image processing unit 130, an image conversion unit 140, an image determination unit 150, a learning-completed model generation unit 160, and a storage unit 170. These components are implemented by executing a program (software) by a hardware processor such as a CPU (Central Processing Unit). Some or all of these components can be implemented by hardware (including a circuit unit: circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), etc., or can be implemented by the cooperation of software and hardware. The program can be pre-stored in a storage device such as an HDD (Hard Disk Drive) or a flash memory (a storage device having a non-temporary storage medium), or can be stored in a removable storage medium such as a DVD or CD-ROM (a non-temporary storage medium), and installed by assembling the storage medium in a drive device. The storage unit 170 is, for example, a HDD, a flash memory, a RAM (Random Access Memory), etc. The storage unit 170 stores, for example, captured image data 172, transformed image data 174, annotation image data 176, annotated image data 178, and a learned model 180. It should be noted that, for the sake of convenience, the image processing device 100 includes a learned model generation unit 160 and a storage unit 170 storing the learned model 180, but the function of generating a learned model and the generated learned model may also be held by a server device different from the image processing device 100.

[0048] The communication unit 110 is an interface for communicating with the communication device 10 of the host vehicle M via the network NW. For example, the communication unit 110 includes a NIC (Network Interface Card), an antenna for wireless communication, and the like.

[0049] The transmission and reception control unit 120 transmits and receives data with the vehicles M1 and M2 and the terminal device 200 using the communication unit 110. More specifically, first, the transmission and reception control unit 120 obtains from the vehicle M1 a plurality of vehicle interior images and vehicle exterior images captured in time series by a camera mounted on the vehicle M1. The time series in this case refers to, for example, capturing images at a predetermined interval (for example, every second) in a driving cycle from the start to the stop of the vehicle M1.

[0050] Figure 3 The diagram shows an example of an in-vehicle image and an out-vehicle image acquired from the vehicle M1. Figure 3 The left part of represents the in-vehicle image obtained from vehicle M1. Figure 3 The right part of represents the vehicle exterior image obtained from vehicle M1. Figure 3 As shown in the left part of FIG. , the in-vehicle image is captured in a state where a camera is provided so as to capture at least the facial area of ​​the driver of the vehicle M1. Figure 3 As shown in the right part of , the vehicle exterior image is captured by a camera installed in a manner that captures at least the front of the vehicle M1 in the traveling direction. The transmission and reception control unit 120 associates the vehicle interior image and vehicle exterior image obtained from the vehicle M1 with the image ID and stores them in the storage unit 170 as captured image data 172.

[0051] Figure 4 1 is a diagram for explaining the processing performed by the image processing unit 130. The image processing unit 130 performs image processing on the captured image data 172, and obtains information such as image attributes, facial attributes, and directions of each image included in the captured image data 172. More specifically, the image processing unit 130 uses a learned model that outputs a classification result indicating whether the image is an in-vehicle image or an out-of-vehicle image when an image is input, and obtains image attributes indicating whether each image included in the captured image data 172 is an in-vehicle image or an out-of-vehicle image.

[0052] The image processing unit 130 uses a learned model that outputs the facial region, the size of the face (the area of ​​the facial region), and the distance from the image shooting position to the face for all faces included in the image when an image is input, to obtain facial attributes of each image included in the captured image data 172. Figure 3As an example, the facial area FA1 of person P1 is obtained from the image inside the vehicle, and the facial area FA2 of person P2, the facial area FA3 of person P3, and the facial area FA4 of person P4 are obtained from the image outside the vehicle. Conveniently, the facial areas FA1, FA2, FA3, and FA4 are obtained as rectangular areas, but the present invention is not limited to such a structure, and for example, a learned model that obtains facial areas along the contour of the person's face may also be used.

[0053] Furthermore, the image processing unit 130 uses a learned model that outputs at least one of a facial direction and a gaze direction as, for example, a vector for all faces included in the image when an image is input, to obtain direction information of faces reflected in each image included in the captured image data 172. More specifically, the image processing unit 130 uses a learned model that outputs a facial direction and a gaze direction for all faces included in the image when an image is input, to obtain direction information of faces reflected in each image included in the captured image data 172 having the attribute of an in-vehicle image. On the other hand, the image processing unit 130 uses a learned model that outputs a facial direction for all faces included in the image when an image is input, to obtain direction information of faces reflected in the captured image data 172 having the attribute of an out-of-vehicle image. This is because, in general, faces reflected in in-vehicle images are closer to the shooting position than in out-of-vehicle images, and there is a higher tendency for faces reflected to be large enough to extract the gaze direction. Figure 3 As an example, the facial direction FD1 and the sight line direction ED1 of the person P1 are obtained from the in-vehicle image, and the facial directions FD2, FD3, and FD4 of the persons P2, P3, and P4 are obtained from the out-vehicle image.

[0054] When the image processing unit 130 obtains image attributes, facial attributes, and direction information about each image of the captured image data 172, it associates the image attributes, facial attributes, and direction information with the image. It should be noted that, as an example, the image processing unit 130 obtains the image attributes, facial attributes, and direction information using the learned model, but the present invention is not limited to such a structure, and the image processing unit 130 may also obtain the image attributes, facial attributes, and direction information using any known method.

[0055] The image conversion unit 140 uses any software that actually has the function of performing the following processing to execute the processing, wherein the processing refers to the processing of replacing the face of the person reflected in each image with the face of another person without changing the direction information of the person reflected in each image in the captured image data 172 processed by the image processing unit 130. Figure 5 1 is a diagram for explaining the processing performed by the image conversion unit 140. Figure 5As shown, the image conversion unit 140 is for Figure 4 The faces of the characters P1, P2 and P3 are replaced with faces of other characters without changing the sight line direction ED1 and the face directions FD1, FD2 and FD3. On the other hand, the face of the character P4 is mosaiced by the image conversion unit 140 and is covered by the mosaic MS.

[0056] That is, the image conversion unit 140 determines whether to replace the face with the face of another person or to perform mosaic processing based on the facial attributes of each face reflected in each image of the captured image data 172. More specifically, the image conversion unit 140 determines whether the size of each face reflected in each image of the captured image data 172 is greater than the first threshold value Th1, and when it is determined that the size of the face is greater than the first threshold value Th1, it determines to replace the face with the face of another person. On the other hand, when it is determined that the size of the face is less than the first threshold value Th1, the image conversion unit 140 determines to perform mosaic processing on the face. Replacing the face of a person reflected in a captured image with the face of another person or performing mosaic processing is an example of "anonymization processing."

[0057] In addition, the image conversion unit 140 determines whether the distance of each face reflected in each image of the captured image data 172 is less than the second threshold value Th2, and when it is determined that the distance of the face is less than the second threshold value Th2, it is determined to replace the face with the face of another person. On the other hand, when it is determined that the distance of the face is greater than the second threshold value Th2, the image conversion unit 140 determines to perform mosaic processing on the face. The image conversion unit 140 repeatedly performs these determination processes according to the number of faces reflected in the image, and replaces each face with the face of another person or performs mosaic processing according to the determination result. The image conversion unit 140 stores the image data obtained by performing such processing on the captured image data 172 as the transformed image data 174 in the storage unit 170. In this way, useful data can be selected as learning data for generating an action prediction model, and the privacy of the person reflected in each image can be protected when the annotator described later performs annotation work.

[0058] It should be noted that at least one of the process of determining whether the size of the face is greater than the first threshold value Th1 and the process of determining whether the distance of the face is less than the second threshold value Th2 may be implemented. In the case of implementing both processes, the image transformation unit 140 may decide to replace the face with the face of another person when the size of the face is greater than the first threshold value Th1 and the distance of the face is less than the second threshold value Th2, or may decide to replace the face with the face of another person when the size of the face is greater than the first threshold value Th1 or the distance of the face is less than the second threshold value Th2.

[0059] Furthermore, the image conversion unit 140 may perform mosaic processing on faces for which direction information has not been successfully acquired among the faces reflected in each image of the captured image data 172 , thereby selecting faces to be used as learning data.

[0060] Figure 6 FIG. 1 is a diagram showing an example of a time series of vehicle interior images converted by the image conversion unit 140 . Figure 6 As an example, the time series of in-car images at three time points, t, t+1, and t+2, are transformed. These time series of in-car images are obtained by photographing the same person and transforming the face, such as Figure 6 As shown in FIG. 1 , the face of the same person may be transformed into the faces of a plurality of different persons according to the operation of the face transformation software. Even if the face of the same person is transformed into the faces of a plurality of different persons, using such transformed image data as learning data directly may deteriorate the accuracy of the action prediction model, which is not preferable. Therefore, the image determination unit 150 determines the continuity of the time series of the in-vehicle image and the out-vehicle image by executing the processing described below.

[0061] Figure 7 1 is a diagram for explaining the processing performed by the image determination unit 150. Figure 7 As shown, the image determination unit 150 first extracts feature points representing the face of the person projected in the transformed image. For example, the image determination unit 150 extracts feature points representing the right eye REP, the left eye LEP, the nose NP, the right corner of the mouth RMP, the left corner of the mouth LMP, and the ear EP from the face of the person projected in the transformed image. The image determination unit 150 extracts feature points of the face of the person tracked as the same person from each of the transformed images in the time series of transformed images, and compares these feature points. It should be noted that whether "tracked as the same person" can be determined by, for example, establishing a correspondence between the same person projected in the captured image in advance at a stage before transforming the image.

[0062] exist Figure 7 In the case of , the image determination unit 150 extracts the feature points of the person reflected in the transformed image at time point t and the feature points of the person reflected in the transformed image at time point t+1. The image determination unit 150 performs a comparison by translating and rotating the two sets of extracted feature points to determine whether they are substantially consistent.

[0063] If the result of the comparison is that the extracted feature points are determined to be substantially consistent, the image determination unit 150 determines that the face of the person tracked as the same person is also the face of the same person after the transformation (that is, there is continuity in the face). On the other hand, if the result of the comparison is that the extracted feature points are determined to be not substantially consistent, the image determination unit 150 determines that the face of the person tracked as the same person is not the face of the same person after the transformation (that is, there is no continuity in the face). In this case, the image transformation unit 140 performs transformation processing again on the face determined to be non-continuous. At this time, the image transformation unit 140 may perform transformation processing again only on the face determined to be non-continuous, or may perform transformation processing again on the faces of all the people reflected in the transformed image of the time series. In addition, for example, the image transformation unit 140 may not perform transformation processing again on the face determined to be non-continuous, but may perform mosaic processing and exclude it from the object used as learning data. In addition, for example, when the image determination unit 150 determines that the face of a person tracked as the same person is not the same person after transformation (i.e., there is no face continuity), the image determination unit 150 may limit the execution of a predetermined process on the transformed images of the time series, i.e., exclude the transformed images of the time series from the objects used as learning data. This can prevent the occurrence of discontinuity caused by the unintended operation of the face transformation software.

[0064] The image determination unit 150 further inputs the transformed image to the above-mentioned learned model that outputs at least one of the facial direction and the sight line direction, and obtains the facial direction FD or the sight line direction ED in the transformed image. The image determination unit 150 determines whether the facial direction FD or the sight line direction ED of the face of the person reflected in the transformed image is substantially consistent with the facial direction FD or the sight line direction ED of the face reflected in the captured image before the transformation. As described above, both the facial direction FD and the sight line direction ED are obtained for the in-vehicle image, and the facial direction FD is obtained for the out-vehicle image. Therefore, the image determination unit 150 determines whether the facial direction FD and the sight line direction ED are substantially consistent between the captured image before the transformation and the transformed image for the in-vehicle image, and determines whether the facial direction FD is substantially consistent between the captured image before the transformation and the transformed image for the out-vehicle image. More specifically, for example, the image determination unit 150 calculates the angle difference between the vector indicating the facial direction FD in the captured image before the transformation and the vector indicating the facial direction FD in the transformed image, and determines that the facial direction FD is substantially consistent when the calculated angle difference is within a threshold value. The same is true for the sight line direction ED. Meeting the continuity of the face or the consistency of directional information is an example of a “prescribed requirement”.

[0065] When it is determined that the face direction FD or the sight line direction ED is not roughly consistent between the captured image before the transformation and the transformed image, the image transformation unit 140 performs transformation processing on the captured image again with respect to the face with which the face direction FD or the sight line direction ED is determined to be not roughly consistent. At this time, the image transformation unit 140 may perform transformation processing again only with respect to the face determined to be not roughly consistent, or may perform transformation processing again with respect to all faces included in the transformed image including the face determined to be not roughly consistent. In addition, for example, the image transformation unit 140 may not perform transformation processing again with respect to the face determined to be not roughly consistent, but may perform mosaic processing, and exclude the face from being used as learning data. In addition, for example, when the result of the determination by the image determination unit 150 is that the face is not roughly consistent, the image determination unit 150 may limit the predetermined processing to be performed on the transformed images in the time series, that is, exclude the transformed images in the time series from being used as learning data. Thus, it is possible to prevent the degradation of information caused by the unintentional operation of the face transformation software.

[0066] It should be noted that, when there are multiple faces reflected in the transformed image (or when the number of faces reflected in the transformed image is greater than or equal to a predetermined value), the determination process related to the continuity of the transformed image and the determination process related to the consistency of the direction information performed by the image determination unit 150 may be performed not on all faces reflected in the transformed image, but only on faces assumed to be of high importance. As an example of faces assumed to be of high importance, the image determination unit 150 may perform these determination processes only on faces whose face size is greater than or equal to a third threshold value Th3 greater than the first threshold value Th1, and may perform these determination processes only on faces whose face distance is less than or equal to a fourth threshold value Th4 less than the second threshold value Th2, in the captured image before the transformation. In addition, for example, the image determination unit 150 may assume that the face of a person who is in front of the vehicle M1 in the traveling direction and the face of a person whose face direction is facing the front of the vehicle M1 in the traveling direction are of high importance, and perform these determination processes. In addition, for example, when the continuity or consistency is denied for a certain face reflected in the transformed image, re-transformation processing may be performed on the face and faces assumed to be highly important.

[0067] When the image determination unit 150 confirms the continuity and consistency of the transformed image of the time series, the transformed image data 174 for which the continuity and consistency are confirmed is stored in the storage unit 170 as the annotation image data 176. At this time, the transformed image data 174 may be stored in the storage unit 170 as the annotation image data 176 together with information indicating the purpose of use, for example, information indicating that the transformed image data 174 is annotation image data for generating a behavior prediction model for predicting the behavior of a person reflected in the input image. The transmission and reception control unit 120 transmits the annotation image data 176 to the terminal device 200. The annotator, who is a user of the terminal device 200, generates annotated image data by performing an annotation operation on the annotation image included in the received annotation image data 176, and transmits the annotated image data to the image processing device 100. The image processing device 100 stores the received annotated image data in the storage unit 170 as the annotated image data 178.

[0068] It should be noted that it is sufficient to execute at least one of the determination processing related to the continuity of the transformed image and the determination processing related to the consistency of the facial direction information performed by the image determination unit 150. Alternatively, when at least one of the continuity and consistency is established, the transformed image data 174 may be stored in the storage unit 170 as the annotation image data 176.

[0069] Furthermore, the image determination unit 150 may, for example, prevent all of the time series captured images (or their transformed images) obtained at a specified interval (for example, every second) in a driving cycle from being stored in the storage unit 170 as annotation image data 176 when there are missing images due to a malfunction of the camera or the like.

[0070] Figure 8 This is a diagram showing an example of annotation work performed by an annotator. Figure 8 The left part of represents the annotation of the transformed image of the in-car image. Figure 8 The right part of represents the annotation of the transformed image of the vehicle exterior image. The annotator gives the transformed image of the vehicle interior image, for example, information indicating whether the direction of the driver's line of sight ED1 reflected in the transformed image is appropriate in the situation shown in the transformed image of the vehicle exterior image at the same time point (for example, 1 if appropriate, and 0 if inappropriate). For example, in Figure 8 In the case of , the transformed image of the exterior image indicates that there is a pedestrian on the left hand side of the vehicle's traveling direction, while the transformed image of the interior image indicates that the driver's line of sight is directed to the left. In other words, it is assumed that the driver paid proper attention to the pedestrian, so the annotator gives appropriate information (i.e., 1) indicating the driver's line of sight ED1.

[0071] Furthermore, the annotator specifies the risk area RA where the person projected in the transformed image is predicted to move, for example, excluding the person subjected to mosaic processing, for the transformed image of the vehicle exterior image. Through the processing performed by the image transformation unit 140 and the image determination unit 150, the face of the person projected in the original image is transformed into the face of another person, so the privacy of the person is protected. At the same time, after the transformation, the facial direction and the line of sight of the person are also maintained, so the annotator can accurately specify the risk area RA while referring to the facial direction and the line of sight of the other person projected in the transformed image. In this way, it is possible to generate learning data effective for learning the machine learning model while protecting the privacy of the person projected in the facial image.

[0072] When the annotated image data 178 is stored in the storage unit 170, the learned model generation unit 160 uses the annotated image data 178 as learning data and generates a learned model using an arbitrary machine learning model. As described above, the learned model is, for example, a model that outputs a predicted action (track) of a person shown in the image outside the vehicle for input of an image outside the vehicle, or a model that urges attention to a pedestrian shown in the image outside the vehicle by considering the driver's line of sight shown in the image inside the vehicle for input of an image inside the vehicle and an image outside the vehicle. The learned model generation unit 160 stores the generated learned model as a learned model 180 in the storage unit 170.

[0073] When generating the learned model 180, the transmission and reception control unit 120 distributes the generated learned model 180 to the vehicle M2 via the network NW. When the vehicle M2 receives the learned model 180, it uses the learned model 180 (more precisely, an application using the learned model 180) to provide driving support to the driver of the vehicle M2.

[0074] Fig. 9 1 is a diagram showing an example of driving assistance using the learned model 180 . Fig. 9 This shows an example in which the vehicle M2 inputs the in-vehicle image and the out-vehicle image captured by the mounted camera to the learned model 180 while driving, and outputs information to the HMI (human machine interface) indicating that the learned model 180 takes into account the driver's line of sight in the in-vehicle image and urges the driver to pay attention to the pedestrian in the out-vehicle image, thereby providing driving support. Fig. 9 As shown, for example, the HMI displays the risk area RA2 corresponding to the pedestrian P5 reflected in the vehicle exterior image, and when the driver's line of sight reflected in the vehicle interior image is not directed toward the pedestrian P5, a warning message ("Please beware of distracted driving") is output as text information or voice information. Thus, driving support that takes the driver's state into consideration can be realized.

[0075] Next, refer to Fig.10 and Fig.11 The flow of processing executed by the image processing apparatus 100 will be described. Fig.10 It is a diagram showing an example of the flow of processing executed by the image conversion unit 140 . Fig.10 The processing shown is executed, for example, at a timing when an in-vehicle image or an out-vehicle image is captured by a camera mounted on the vehicle M1 and processed by the image processing unit 130 .

[0076] First, the image conversion unit 140 acquires a captured image included in the captured image data 172 that has been processed by the image processing unit 130 (step S100 ). Next, the image conversion unit 140 selects one face reflected in the acquired captured image (step S102 ).

[0077] Next, the image conversion unit 140 determines whether the size of the selected face is greater than or equal to the first threshold value Th1 (step S104). If it is determined that the size of the selected face is greater than or equal to the first threshold value Th1, the image conversion unit 140 converts the face into a face of another person (step S106). On the other hand, if it is determined that the size of the selected face is less than the first threshold value Th1, the image conversion unit 140 next determines whether the distance of the selected face is less than or equal to the second threshold value Th2 (step S108).

[0078] When it is determined that the distance of the selected face is less than the second threshold value Th2, the image conversion unit 140 proceeds to step S106 and converts the face into the face of another person. On the other hand, when it is determined that the distance of the selected face is greater than the second threshold value Th2, the image conversion unit 140 performs mosaic processing on the face (step S110). Next, the image conversion unit 140 determines whether the processing has been performed on all the faces reflected in the acquired captured image (step S112).

[0079] When it is determined that the processing has been performed on all the faces reflected in the acquired captured image, the image conversion unit 140 obtains the image obtained by performing the processing on all the faces as a converted image, and stores it in the storage unit 170 as converted image data 174 (step S114). On the other hand, when it is determined that the processing has not been performed on all the faces reflected in the acquired captured image, the image conversion unit 140 returns the processing to step S102. Thus, the processing of this flowchart ends.

[0080] Fig.11 It is a diagram showing an example of the flow of processing executed by the image determination unit 150 . Fig.11The processing shown is executed, for example, at a timing when a time-series converted image is obtained by performing the above-mentioned conversion processing on a time-series captured image captured in one driving cycle from start to stop of the vehicle M1 .

[0081] First, the image determination unit 150 acquires time-series transformed images (step S200 ). Next, the image determination unit 150 selects the face of a person tracked as the same person before transformation from among the acquired time-series transformed images (step S202 ).

[0082] Next, the image determination unit 150 extracts feature points from the faces of the persons tracked as the same person before the transformation in each of the transformed images in the time series, and compares them to determine whether these faces are the same face after the transformation (step S204). If it is determined that the faces are the same face after the transformation, the image determination unit 150 next determines whether the obtained transformed images in the time series are in-vehicle images (step S206). On the other hand, if it is determined that the faces are not the same face, the image determination unit 150 causes the image transformation unit 140 to transform again the faces of the persons tracked as the same person before the transformation in the captured images in the time series (step S208). Thereafter, the image determination unit 150 performs the processing of step S204 again on the re-transformed faces.

[0083] If it is determined in step S206 that the acquired time series transformed images are in-vehicle images, the image determination unit 150 determines whether the sight lines and facial directions of these faces are consistent with the images before transformation (step S210). On the other hand, if it is determined that the acquired time series transformed images are not in-vehicle images, that is, the acquired time series transformed images are out-vehicle images, the image determination unit 150 determines whether the facial directions of these faces are consistent with the images before transformation (step S212). If it is determined that they are inconsistent in the processing of step S210 or step S212, the image determination unit 150 advances the processing to step S208.

[0084] If it is determined that the faces are matched in the process of step S210 or step S212, the image determination unit 150 determines that the faces are normally transformed, and determines whether the process is performed on all the faces reflected in the time series transformed images (step S214). If it is determined that the process is performed on all the faces reflected in the time series transformed images, the image determination unit 150 obtains the time series transformed images as annotation images, and causes the transmission and reception control unit 120 to send the obtained annotation images to the terminal device 200 (step S216). On the other hand, if it is determined that the process is not performed on all the faces reflected in the time series transformed images, the image determination unit 150 returns the process to step S202. Thus, the process of this flowchart ends.

[0085] According to the present embodiment described above, when it is determined that a plurality of input images subjected to anonymization processing satisfy a prescribed requirement, a prescribed processing is performed on the plurality of input images subjected to anonymization processing, and the anonymization processing includes a processing of changing the face of a person shown in the plurality of input images to the face of another person, and the prescribed requirement includes that the face of a person tracked as the same person shown in the plurality of input images subjected to anonymization processing is the face of the same person obtained by the anonymization processing. That is, in the present embodiment, the face of the same person before the anonymization processing is guaranteed to be the face of the same person during the anonymization processing, and is used as learning data. Thus, it is possible to generate learning data effective for learning a machine learning model while protecting the privacy of the person shown in the facial image.

[0086] In addition, according to this embodiment, the prescribed requirement includes that the direction information of the face of a person tracked as the same person in a plurality of input images is consistent with the direction information of the face of the same person in the plurality of input images to which the anonymization process is applied. That is, in this embodiment, it is ensured that the direction information of the face of the same person remains unchanged even after the anonymization process is applied. Thus, it is possible to generate learning data effective for learning a machine learning model while protecting the privacy of the person reflected in the facial image.

[0087] In addition, according to the present embodiment, the prescribed requirements are determined based on the image attributes that are the shooting methods of the plurality of input images. That is, in the present embodiment, the prescribed processing is performed, which refers to the processing that considers the shooting methods of each of the plurality of input images, for example, is saved as learning information for generating an action prediction model. In this way, it is possible to generate learning data that is effective for learning a machine learning model while protecting the privacy of the person reflected in the facial image.

[0088] In addition, according to the present embodiment, it is determined whether to perform anonymization processing by a first method or by a second method different from the first method based on the size of the face reflected in each of the multiple input images or the distance from the shooting location to the face. That is, in the present embodiment, the method of anonymization processing performed on the face is changed depending on whether it is useful for learning the machine learning model. In this way, it is possible to generate learning data that is effective for learning the machine learning model while protecting the privacy of the person reflected in the facial image.

[0089] [Variation example]

[0090] As described above, in this embodiment, an example is described in which, when the image determination unit 150 determines that the face reflected in the transformed image does not satisfy the prescribed requirements, the transformed image is re-transformed and mosaic processing is performed. However, when the image determination unit 150 determines that the prescribed requirements are not satisfied, the image determination unit 150 may perform processing such as not performing the prescribed processing on the transformed image, that is, limiting the execution of the prescribed processing (not storing the image, not sending it to the server, etc.).

[0091] Furthermore, in the present embodiment, an example is described in which the image processing device 100 is actually installed as a server device separate from the vehicle M1. However, as a variation of the present embodiment, the image processing device 100, more specifically, a device having at least the functions of the image processing unit 130, the image conversion unit 140, and the image determination unit 150, may also be mounted on the vehicle M1 as a vehicle-mounted device. In this case, the vehicle-mounted device performs the above-mentioned processing by the image processing unit 130 on the image captured by the vehicle-mounted camera, performs anonymization by the image conversion unit 140, and performs determination by the image determination unit 150. Thereafter, the vehicle-mounted device sends the anonymized image in which the continuity of the face and the consistency of the direction information are confirmed by the image determination unit 150 to the external image server.

[0092] When the image server receives the anonymized image from the vehicle M1, it stores the received anonymized image as annotation image data in the storage unit, and sends the annotation image data to the terminal device 200 of the annotator, or permits the terminal device 200 to access the annotation image data. When the image server receives the annotated image data from the terminal device 200, it generates the learned model 180 based on the annotated image data, and distributes the generated learned model 180 to the vehicle M2. In this way, as in the present embodiment, it is possible to generate learning data effective for learning the machine learning model while protecting the privacy of the person reflected in the facial image. Moreover, according to this variant, the vehicle-mounted device sends the anonymized image to the image server after anonymizing the image, so that the privacy of the person reflected in the facial image can be further reliably protected.

[0093] Furthermore, as another embodiment, the vehicle-mounted device may have only some of the functions of the image processing unit 130, the image conversion unit 140, and the image determination unit 150, and the image server may have the remaining functions. For example, the vehicle-mounted device may have the functions of the image processing unit 130 and the image conversion unit 140, and the image server may have the function of the image determination unit 150, or the vehicle-mounted device may have the functions of the image processing unit 130, and the image server may have the functions of the image conversion unit 140 and the image determination unit 150.

[0094] The above-described embodiment can be expressed as follows.

[0095] An image processing device, wherein:

[0096] The image processing device is configured to include:

[0097] a storage medium that stores computer-readable instructions; and

[0098] a processor connected to the storage medium,

[0099] The processor executes the computer-readable instructions to:

[0100] Obtaining an image attribute, wherein the image attribute is a shooting method of an input image;

[0101] Anonymizing the input image;

[0102] determining whether the input image subjected to the anonymization process satisfies a prescribed requirement; and

[0103] When it is determined that the input image subjected to the anonymization process satisfies the predetermined requirement, performing predetermined processing on the input image subjected to the anonymization process;

[0104] The predetermined requirement is determined according to the acquired image attribute.

[0105] Although specific embodiments of the present invention have been described above using the embodiments, the present invention is not limited to such embodiments at all, and various modifications and substitutions can be made without departing from the gist of the present invention.

[0106] Description of reference numerals:

[0107] 100 Image processing device

[0108] 110 Ministry of Communications

[0109] 120 Transceiver Control Unit

[0110] 130 Image Processing Unit

[0111] 140 Image conversion unit

[0112] 150 Image determination unit

[0113] 160 Learning Complete Model Generation Section

[0114] 170 Storage

[0115] 172 captured image data

[0116] 174 Transform Image Data

[0117] 176 Annotation image data

[0118] 178 annotated image data

[0119] 180 Learning completed model.

Claims

1. An image processing device, wherein: The image processing device comprises: an image attribute acquisition unit that acquires an image attribute, wherein the image attribute is a shooting method of an input image; An image conversion unit, which performs anonymization processing on the input image; as well as an image determination unit for determining whether the input image subjected to the anonymization process satisfies a predetermined requirement, When the image determination unit determines that the input image subjected to the anonymization process satisfies the predetermined requirement, the image determination unit performs a predetermined process on the input image subjected to the anonymization process. The predetermined requirement is determined according to the acquired image attribute.

2. The image processing device according to claim 1, wherein: The predetermined process is a process of storing the input image subjected to the anonymization process as a target image for annotation work.

3. The image processing device according to claim 1, wherein: The predetermined processing is processing of storing the input image subjected to the anonymization processing as learning information for generating a behavior prediction model, the behavior prediction model being a model for predicting the behavior of a person reflected in the input image.

4. The image processing device according to claim 1, wherein: The predetermined process is a process of transmitting the input image subjected to the anonymization process to an image server via a communication means.

5. The image processing device according to claim 1, wherein: The image attribute is information indicating at least whether the input image is an image captured from the inside of a vehicle or an image captured from the outside of the vehicle, the vehicle being equipped with a camera that captured the input image.

6. The image processing device according to claim 5, wherein: The anonymization process includes a process of changing the face of a person reflected in the input image to the face of another person. The predetermined requirement includes, when the image attribute indicates that the image is an image of the interior of the vehicle, whether the gaze direction of the face of the person matches the gaze direction of the face of the other person.

7. The image processing device according to claim 6, wherein: The prescribed requirements include, when the image attribute indicates that the image is taken of the interior of the vehicle, whether the gaze direction of the face of the person is consistent with the gaze direction of the face of the other person, and whether the facial direction of the face of the person is consistent with the facial direction of the face of the other person, The predetermined requirement includes, when the image attribute indicates that the image is an image of the exterior of the vehicle, whether the facial direction of the face of the person matches the facial direction of the face of the other person.

8. The image processing device according to claim 6, wherein: The prescribed requirement does not include whether the gaze direction of the face of the person matches the gaze direction of the face of the other person, when the image attribute indicates that the image is an image of the outside of the vehicle.

9. The image processing apparatus according to claim 1, wherein: The image conversion unit performs the anonymization process on the input image again when the image determination unit determines that the input image on which the anonymization process has been performed does not satisfy the predetermined requirement.

10. The image processing apparatus according to claim 1, wherein: The image conversion unit does not perform the predetermined process on the input image that has been subjected to the anonymization process, when the image determination unit determines that the input image that has been subjected to the anonymization process does not satisfy the predetermined requirement.

11. An image processing system, wherein: The image processing system comprises: an image attribute acquisition unit that acquires an image attribute, wherein the image attribute is a shooting method of an input image; An image conversion unit, which performs anonymization processing on the input image; as well as an image determination unit for determining whether the input image subjected to the anonymization process satisfies a predetermined requirement, When the image determination unit determines that the input image subjected to the anonymization process satisfies the predetermined requirement, the image determination unit performs a predetermined process on the input image subjected to the anonymization process. The predetermined requirement is determined according to the acquired image attribute.

12. An image processing method, wherein: The image processing method enables the computer to perform the following processing: Obtaining an image attribute, wherein the image attribute is a shooting method of an input image; Anonymizing the input image; determining whether the input image subjected to the anonymization process satisfies a prescribed requirement; as well as When it is determined that the input image subjected to the anonymization process satisfies the predetermined requirement, performing predetermined processing on the input image subjected to the anonymization process; The predetermined requirement is determined according to the acquired image attribute.

13. A program, wherein: The program causes the computer to perform the following processing: Obtaining an image attribute, wherein the image attribute is a shooting method of an input image; Anonymizing the input image; determining whether the input image subjected to the anonymization process satisfies a prescribed requirement; as well as When it is determined that the input image subjected to the anonymization process satisfies the predetermined requirement, performing predetermined processing on the input image subjected to the anonymization process; The predetermined requirement is determined according to the acquired image attribute.

Images