Image processing apparatus, method for processing images, and image processing program

Abstract

To appropriately control settings for performing mask processing on an image.SOLUTION: An image processing apparatus detects a target region in which an object to be protected as personal information is imaged, from an image output from an imaging device mounted in an own vehicle, specifies a driving scene according to the traveling lane and the vehicle speed of the own vehicle, determines a coefficient representing the strength of mask processing according to the driving scene for each divided region of the image, and performs mask processing on the target region according to the coefficient.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program for performing mask processing on an object to be protected from personal information captured in an image. [Background technology]

[0002] A technology has been proposed for detecting areas corresponding to personal information, such as a person's face, from an image recorded by a drive recorder or the like, and concealing the information by performing blurring or other processing (Patent Document 1). In this technology, a detection area, which is an area in which a detection target object is captured, is detected from an image captured by a camera mounted on a vehicle. A threshold is then set based on the vehicle's driving conditions, the type of object, and the position of the detection area in the image, and it is determined whether the degree of certainty at the time of detection of the detection area is equal to or greater than the threshold. The detection result is used in masking processing to conceal the object for the purpose of protecting personal information, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-020507 Summary of the Invention [Problem to be solved by the invention]

[0004] When a predetermined fixed blur coefficient (in other words, blurring of standard strength) is applied to a case where a detected object appears relatively small in an image and a case where the object appears relatively large, for example, blurring may be insufficient when the object appears relatively large. Therefore, for example, when the object is a person, blurring may not be sufficient to conceal personal information. Therefore, an object of the present disclosure is to provide a technology for appropriately controlling settings for performing masking processing on an image. [Means for solving the problem]

[0005] The image processing device according to the present disclosure detects a target area in which an object for protecting personal information is captured from an image output by an imaging device mounted on the vehicle, identifies a driving scene according to the lane and vehicle speed of the vehicle, determines a coefficient representing the strength of mask processing according to the driving scene for each divided area into which the image is divided, and performs mask processing on the target area according to the coefficient. [Effects of the Invention]

[0006] According to the present technology, it is possible to appropriately control settings for performing mask processing on an image. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram for explaining an outline of the embodiment. [Figure 2] FIG. 2 is a diagram for explaining the strength of the masking process. [Figure 3] FIG. 3 is a block diagram illustrating an example of a system configuration. [Figure 4] FIG. 4 is a diagram showing an example of image data captured by a camera. [Figure 5] FIG. 5 is a processing flow diagram showing an example of the concealment processing. [Figure 6] FIG. 6 is a diagram showing an example of data defining the relationship between a driving scene and the strength of blurring. [Figure 7] FIG. 7 is a diagram for explaining the blur coefficient. [Figure 8] FIG. 8 is a processing flow diagram showing an example of the concealment processing. [Figure 9] FIG. 9 is a diagram showing another example of image data captured by a camera. DETAILED DESCRIPTION OF THE INVENTION

[0008] <Embodiment> Hereinafter, an embodiment will be described with reference to the drawings. FIG. 1 is a diagram for explaining an overview of the embodiment. In this embodiment, a person whose personal information is to be protected is detected from a video captured by a drive recorder or the like installed in a vehicle, and a masking process is performed to apply a blur or other mask to the face. Specifically, the in-vehicle device performs object detection on the input video using a known method (step S1) and obtains the position, size, and reliability score of the detected object. The in-vehicle device then determines whether the object detection result is valid by, for example, comparing the reliability score with a predetermined threshold (step S2). If the result is valid, the in-vehicle device performs the masking process on the position and size information indicating the area in which the object is captured. The in-vehicle device then identifies the lane in which the vehicle is located by detecting white lines in the video (step S3) and obtains the detection result of the driving lane. Note that the lane identification may use the vehicle's position information, map information, etc. instead of or in addition to image processing. The in-vehicle device also obtains information indicating the vehicle's speed from a vehicle speed sensor. Then, in the concealment process (step S4), the driving scene is identified based on the lane detection result and the vehicle speed, and the masking process settings are changed according to the driving scene. Note that the process shown in Fig. 1 is an example, and the order of the processes is not limited to the example shown in Fig. 1.

[0009] FIG. 2 is a diagram illustrating the strength of masking. In the example of FIG. 2, a person's face is masked by blurring. A person located some distance from the vehicle appears relatively small in the video, so standard blurring can adequately conceal the person. However, a person located close to the vehicle appears large in the image, so standard blurring may not provide sufficient concealment. (A) on the left of FIG. 2 is an example of an image of a person when standard blurring is applied. In the example of FIG. 2(A), the blurring strength is insufficient compared to the size of the person in the image data, which may result in the person being identified. (B) on the right of FIG. 2 is an image of a person when stronger blurring than standard is applied. As shown in FIG. 2(B), applying strong blurring according to the size of the object makes it possible to adequately conceal the person's face.

[0010] In this embodiment, in step S4 shown in Fig. 1, masking is performed in accordance with the size of the detected object in areas where a person is likely to appear large, depending on the predefined driving scene, and masking with a standard strength is performed in areas where a person is unlikely to appear large. In this way, the settings for performing masking on the image can be appropriately controlled.

[0011] FIG. 3 is a block diagram illustrating an example of the system configuration. The vehicle 1 is, for example, an automobile. The vehicle 1 includes an in-vehicle device 10, a camera 20, a vehicle speed sensor 30, and a position sensor 40. The in-vehicle device 10 is a computer, and functions as, for example, a drive recorder in cooperation with the camera 20. The in-vehicle device 10 according to this embodiment performs object detection, targeting human faces in captured images, and performs mask processing on the detected objects. Preferably, the in-vehicle device 10 can identify the position of the vehicle 1 on a map based on position information indicating the position of the vehicle 1. The in-vehicle device 10 may be, for example, a device that doubles as part of a car navigation system or a device that cooperates with a car navigation system. The in-vehicle device 10 includes a processor 11, a storage device 12, and a user interface (UI) 13.

[0012] The processor 11 is an arithmetic processing device such as a CPU (Central Processing Unit). The processor 11 executes a program to perform various processes according to the embodiment. In this embodiment, the processor 11 detects an area where personal information is captured from image data output by the camera 20, and creates anonymized data by masking the detected area. In this embodiment, the strength of the mask is changed based on the driving scene, which is determined depending on the lane in which the host vehicle is located and whether the host vehicle is currently traveling. The storage device 12 includes at least one of a main storage device such as a random access memory (RAM) or a read-only memory (ROM), and an auxiliary storage device such as a hard-disk drive (HDD), a solid-state drive (SSD), or a flash memory. The storage device 12 temporarily stores programs read by the processor 11 and information to be processed, and secures a working area for the processor 11. The storage device 12 also temporarily or permanently stores image data output by the camera 20 and anonymized data obtained by masking a portion of the image data. The UI 13 is a user interface, such as a display with a stacked touch panel, a display and physical operation buttons, or a combination of a display, a touch panel, and operation buttons. The processor 11 outputs, for example, the anonymized data via the UI 13.

[0013] The camera 20 is an imaging device that converts light into an electrical signal using an image sensor such as a CCD or CMOS, and creates and outputs image data. FIG. 4 is a diagram showing an example of image data captured by the camera. The camera 20 outputs an image 200, for example, captured in the direction of travel of the vehicle 1. The image 200 corresponds to, for example, one still image among a series of frames constituting a moving image. That is, the camera 20 repeats a process of creating and outputting the image 200. Note that the focal length, etc., of the camera 20 are not particularly limited, and the angle of view of the image 200 is not limited to the example shown in FIG. 4. The camera 20 may also be a so-called omnidirectional camera. For example, the camera 20 may include two or more imaging devices, and multiple images created by the camera 20 may be output as is, or may be stitched together to output a single image. Note that areas 201, 202, etc. detected from the image 200 will be described later.

[0014] The vehicle speed sensor 30 in FIG. 3 is, for example, a vehicle speed pulse generator, which generates a pulse signal (vehicle speed signal) corresponding to the rotation of the axle. The vehicle speed signal is transmitted to an ECU (Electronic Control Unit). The vehicle speed signal is transmitted to the in-vehicle device 10 via the vehicle speed sensor 11. The in-vehicle device 10 can determine, for example, whether the vehicle is moving or stopped, based on the vehicle speed signal.

[0015] The position sensor 40 is a receiver that receives signals from satellites in a global navigation satellite system (GNSS), calculates coordinates indicating the position of the position sensor 40, and outputs the coordinates as position information.

[0016] <Concealment processing> FIG. 5 is a processing flow diagram showing an example of the concealment processing. The in-vehicle device 10 starts the concealment processing in conjunction with, for example, the start of image capture by the camera 20. The camera 20 continuously stores the captured image data in the storage device 12 of the in-vehicle device 10. Note that an object detection model for detecting an object from the image data is assumed to be stored in advance in the storage device 12. The object detection model can be created by a machine learning method such as deep learning using a convolutional neural network. In addition, a lane detection model for detecting, for example, white lines on a road from the image data and identifying the position of the lane on which the host vehicle is traveling (e.g., the number of lanes from the roadside) is also assumed to be stored in advance in the storage device 12.

[0017] The processor 11 of the in-vehicle device 10 acquires image data output by the camera 20 (FIG. 5: step S11). The image data is data corresponding to one frame of a moving image output by the camera 20. In this step, for example, an image such as that shown in FIG. 4 is read from the storage device 12.

[0018] After step S11, the processor 11 identifies the lane in which the host vehicle is traveling from the image data (FIG. 5: step S12). In this step, the processor 11 uses an existing white line detection technology to detect, for example, the position of the lane in which the host vehicle is traveling. For example, when the vehicle is traveling in a lane that includes multiple lanes, When both vehicles 1 are traveling, the order of the lanes in which the vehicle 1 is traveling is determined, for example, starting from the leftmost lane (in other words, the lane closest to the road). Note that instead of or in addition to detecting white lines by image processing, processor 11 may identify the lane in which the vehicle 1 is traveling using position information output by position sensor 40. Processor 11 may also perform the processing of this step when it determines, using map information and position information output by position sensor 40, that the road on which vehicle 1 is traveling includes multiple lanes on one side. Processor 11 may also determine that the vehicle 1 is traveling in the leftmost lane even on a road with one lane in each direction, a road with only one lane without a center line, a one-way road, etc.

[0019] After step S12, the processor 11 acquires the vehicle speed of the vehicle 1 (FIG. 5: S13). In this step, the processor 11 acquires a vehicle speed signal from, for example, the vehicle speed sensor 30. Whether the vehicle 1 is stopped or moving can be determined based on whether the vehicle speed signal is, for example, zero.

[0020] After S13, the processor 11 identifies the driving scene (FIG. 5: S14). In this step, the processor 11 identifies the driving scene of the vehicle 1 that corresponds to the current time of processing from among a plurality of predetermined driving scenes. The driving scene is determined based on the lane in which the vehicle 1 is located and whether the vehicle 1 is currently driving.

[0021] FIG. 6 is a diagram showing an example of data defining the relationship between a driving scene and the strength of blurring applied to an area in an image. The table in FIG. 6 includes attributes of an identification number ("No."), a "driving scene," and a "blurring strength." The "driving scene" includes columns of "lane determination" and "driving determination." The "blurring strength" includes columns of "left," "front," and "right." The "No." field stores identification information for uniquely identifying a record. The "lane determination" field stores the position of the lane in which the host vehicle is traveling, which was identified in step S12, in order from the leftmost lane, for example. The "driving determination" field stores information indicating whether the host vehicle is stopped or traveling, based on the vehicle speed acquired in step S13. In step S14, the processor 11 determines which of the driving scenes determined according to the combination of the lane determination and the driving determination corresponds to the current situation of the vehicle 1. For example, in a lane with three lanes on each side, it is determined whether the number corresponds to any of records 1 to 6. In a lane with two lanes on each side, it is determined whether the number corresponds to any of records 1 to 4. In a road with one lane on each side, a road with only one lane without a center line, a one-way road, etc., it is determined whether the number corresponds to any of records 1 or 2.

[0022] After step S14, processor 11 segments the image according to the driving scene (FIG. 5: step S15). In this step, areas are segmented to which different settings for masking are applied. That is, areas to which the same settings should be applied are determined based on information registered in the "blur strength" field in FIG. 6, for example. In the example of FIG. 6, the "blur strength" field registers information indicating the strength of blur to be applied to each of three areas, "left," "center," and "right." The three areas correspond to, for example, the left, center, and right areas indicated by the dashed rectangles in FIG. 4. When the blur strength is "standard," a relatively weak blur is applied that is determined in advance. When the blur strength is set "according to size," the blur strength is changed according to the size of the object captured in the image.

[0023] Here, the strength of the blurring is, for example, the size of the filter (the size of the range in which the pixel colors in the image data are averaged), and the parameter for changing the strength of the blurring is called the "blurring coefficient" or "masking coefficient" in this embodiment. For example, when blurring is performed using a Gaussian filter, the blurring coefficient is the radius (the radius of the convolution kernel). Depending on the blurring coefficient, the strength of the blurring can be changed as shown in Figure 2. do.

[0024] FIG. 7 is a diagram illustrating the blur coefficient when the blur strength is set "according to size." In the graph of FIG. 7, the vertical axis represents the blur coefficient (y) and the horizontal axis represents the size (x) of the target of the concealment process. The blur coefficient y increases in proportion to the target size x and can be calculated, for example, by the formula ax+b. The slope a and offset b are positive values that are set appropriately. Note that the blur coefficient when the blur strength is set "according to size" is set to be larger than the blur coefficient when the blur strength is "standard." Therefore, the slope a and offset b are determined so that the blur coefficient y when the target size x is zero, or the blur coefficient y at the size of the target detection limit (the size of the smallest target that can be detected) in step S17 described below, is equal to or larger than the blur coefficient when the blur strength is "standard."

[0025] For example, record No. 1 in FIG. 6 indicates that the blur strength is set "according to size" for objects detected in the left and front regions, and that the blur strength setting is "standard" for objects detected in the right region. Therefore, in step S15, the image is divided into a left and center region and a right region. Record No. 2 indicates that the blur strength is set "according to size" for the left region, and that the blur strength setting is "standard" for the front and right regions. Therefore, in step S15, the image is divided into a left region, a front and right region. Records No. 3 and 5 indicate that the blur strength is set "standard" for the left and right regions, and that the blur strength setting is "according to size" for the center region. Therefore, in step S15, the image is divided into a left and right region and a center region. Records No. 4 and 6 indicate that the blur strength setting is "standard" for all three regions. Therefore, in step S15, the entire area is partitioned into one region.

[0026] After step S15, processor 11 determines a coefficient to be applied to each section according to the driving scene (FIG. 5: step S16). In this step, the blur coefficient is determined for each region in the divided image according to the information registered in each field of "blur strength" shown in FIG.

[0027] After step S16, the processor 11 detects one object from the image (FIG. 8: step S17). In this step, the object is detected from the image based on an existing object detection method. In the example of FIG. 4, for example, one of areas 201 and 202 containing a person's face is detected depending on the scanning order in this step. The processor 11 also acquires information about a bounding box representing the area in which the object is detected (i.e., coordinates representing the position in the image, and width and height representing the size of the area), as well as a score representing the reliability of the detection (reliability score). The reliability score is, for example, an existing index value calculated based on an index representing whether the bounding box accurately surrounds the object and an index representing the accuracy of the identified object.

[0028] After step S17, the processor 11 determines whether the reliability of the object detection is equal to or greater than a threshold (FIG. 8: step S18). In this step, it is determined whether the reliability score acquired in step S17 is equal to or greater than a predetermined threshold.

[0029] If it is determined in step S18 that the reliability is equal to or greater than the threshold (step S18: YES), processor 11 performs blurring on the detected object based on the blur coefficient set in step S16 (FIG. 8: step S19). In this step, the bounding box detected in step S17 is blurred to a strength that corresponds to the driving scene set in step S16.

[0030] When the vehicle is traveling in the leftmost lane, which is on the side of the road in the direction of travel, there is a possibility that pedestrians or other objects will appear in the left area of the image captured by the camera. Therefore, objects detected in the left area, whether the vehicle is traveling or stopped, are blurred according to their size (Figure 6: Nos. 1 and 2). Even when traveling in a lane other than the leftmost lane (Figure 6: Nos. 3-6), if vehicle 1 is stopped at a traffic light as shown in Figure 8, there is a possibility that an object such as a pedestrian will cross just in front of the vehicle and appear relatively large in the image. Therefore, when the vehicle is stopped, objects detected in the area in front of the vehicle are blurred according to their size (Figure 6: Nos. 3 and 5). Note that the areas in Figure 6 where the blur strength is registered as "standard" correspond to patterns where there is a low possibility that an object will appear large in that area during that driving scene.

[0031] FIG. 9 is a diagram showing another example of image data captured by a camera. In the example of FIG. 9, it is assumed that in step S14 of FIG. 5, the host vehicle is in the leftmost lane and is determined to be stopped, and in step S16, the blur coefficient defined in record No. 1 of FIG. 6 is set. The image of FIG. 9 is partitioned into left and center regions and a right region. Furthermore, for objects detected in the left and center regions, the strength of blur is set "according to size." For objects detected in the right region, the strength of blur is set to "standard."

[0032] After step S19 in Fig. 8, or if it is determined in step S18 that the reliability is not equal to or greater than the threshold (step S18: NO), processor 11 determines whether an unprocessed object exists in the image (Fig. 8: step S20). If it is determined that an unprocessed object exists (step S20: YES), processor 11 returns to step S7 and repeats the process. For example, if one of areas 201 and 202 containing a person's face in Fig. 4 is unprocessed, the processes from steps S17 to S19 are performed on the unprocessed object.

[0033] On the other hand, if it is determined that there is no unprocessed object (step S20: NO), the processor 11 outputs the image that has been subjected to the concealment process (FIG. 8: step S21). In this step, an image in which the detected object is blurred is displayed on the UI 13 of the in-vehicle device 10. For example, an image in which the detected object is blurred according to the driving scene is displayed on the display, in contrast to the images shown in FIG. 4 and FIG. 9.

[0034] After step S21, the processor 11 determines whether there is other image data to be processed (FIG. 8: step S22). In this step, it is determined whether there is image data corresponding to the next frame in the video. If there is next image data to be processed (step S22: YES), the processor 11 returns to S11 in FIG. 5 and repeats the process. On the other hand, if there is no next image data (step S22: NO), the processor 11 ends the concealment process. For example, if the camera 20 stops the imaging process, it is determined that there is no next image data.

[0035] <Effects> In this embodiment, by setting a blur coefficient according to the driving scene, it is possible to appropriately conceal even relatively large objects. A driving scene can be grasped in a manner suitable for this embodiment by combining the determination result of the vehicle's driving lane and the determination result of whether the vehicle is moving. That is, it is possible to partition areas in the image where large objects may appear depending on the situation. Furthermore, the more objects detected, the greater the load of the concealment process. By partitioning the image into areas according to the driving scene and changing the settings for masking process for each area, it is possible to suppress an increase in the load required for blurring process, especially when many objects are detected. That is, it is possible to appropriately control the settings for masking the image.

[0036] <Modification> The driving scene may be determined based on only one of the lane in which the vehicle is located and whether the vehicle is moving or stopped. The driving scene may also be further subdivided by combining other determinations.

[0037] Furthermore, the division of the image is not limited to three regions, left, center, and right. For example, it may be divided into two regions, left and right, or into four or more regions. Furthermore, the image may also be divided in the vertical direction.

[0038] Furthermore, the masking process may be a mosaic process. In the mosaic process, for example, the image is concealed by filling in the color of the pixels contained in the image with a color that is averaged for each range of a predetermined size. In this case, too, if the object captured in the image is relatively large, concealment may be insufficient. Therefore, for each driving scene, in an area in the image where a person is likely to appear large, the range in which the pixels are averaged may be increased according to the size of the detected object. The parameter that determines the size of the range is a coefficient of the mosaic process, which corresponds to the "coefficient of the masking process" described above.

[0039] The timing for performing the anonymization process is not limited to immediately after the camera 20 outputs the image data. For example, the anonymization process may be performed on the read image data when the video data stored in the storage device 12 or a storage on the network is played back.

[0040] Furthermore, if the same object moves across the region partitioned in step S5 between frames of a moving image, the masking coefficients may be smoothly changed. The same object in multiple frames can be tracked using a known tracking process. For example, in step S7 of FIG. 8, processor 11 tracks the object detected in the previous image (frame) and detects any newly found objects. For each object masked in the previous image, processor 11 stores the masking coefficients in at least the most recent frame in storage device 12, and determines in step S9 whether the masking coefficients to be applied are the same as those in the previous frame. If the masking coefficients (first coefficients) in the previous frame differ from the new masking coefficients (second coefficients), processor 11 does not immediately change from the first coefficients to the second coefficients, but instead applies masking to the image by gradually changing the masking coefficients from the first coefficients to the second coefficients over a predetermined number of subsequent frames. In this way, for example, the strength of the masking applied to a person who moves from the right area to the left area in Fig. 9 changes gradually rather than suddenly at the boundary between the areas, thereby reducing the sense of discomfort felt by the user watching the video due to the change in the strength of the masking.

[0041] <Other> The illustrated system and device configurations are merely examples and are not limited to the above examples. For example, at least a portion of the processing performed by the on-board device 10 of the vehicle 1 may be shared and executed by multiple devices, or may be executed in parallel by multiple devices. Similarly, the processing shown in the processing flow may be executed in a different order or in parallel, as long as the results are unchanged. For example, the processing of steps S2 and S3 may be executed in the reverse order. Furthermore, the processing of steps S2 and S3 and the processing of steps S5 and S6 may be executed in parallel.

[0042] The present invention also includes a computer program for executing the above-described processing method, and a computer-readable recording medium on which the program is recorded. The above-described processing can be performed by loading the recording medium into a computer and executing the recorded program. A computer-readable recording medium is a medium that stores information such as data and programs electronically. "A recording medium is a storage medium that stores information magnetically, optically, mechanically, or chemically and can be read by a computer. Among such storage media, those that can be removed from a computer include flexible disks, magneto-optical disks, optical disks, magnetic tapes, memory cards, etc. Furthermore, storage media that are fixed to a computer include hard disk drives and ROMs. [Explanation of symbols]

[0043] 1: Vehicle, 10: In-vehicle device, 11: Processor, 12: Storage device, 13: User interface (UI), 20: Camera, 30: Vehicle speed sensor, 40: Position sensor

Claims

1. Detecting a target area in which an object to be protected for personal information is captured from an image output by an imaging device mounted on the vehicle; Identifying a driving scene according to the driving lane and vehicle speed of the host vehicle; determining a coefficient representing the strength of mask processing for each divided area of the image in accordance with the driving scene; A mask process is performed on the target region according to the coefficient. Image processing device.

2. estimating the driving lane based on at least one of image recognition based on the image and position information output by a position sensor mounted on the vehicle; The vehicle speed is acquired based on information detected by a vehicle speed sensor mounted on the vehicle. The image processing device according to claim 1 .

3. The divided area is divided according to the driving scene. The image processing device according to claim 1 .

4. The coefficients are determined so that the larger the area in the image where the object to be protected is captured, the stronger the mask. The image processing device according to claim 1 .

5. tracking the object to be protected in a plurality of images constituting a video sequence; When it is determined that the same protection object has moved from a partitioned area to which a first coefficient is applied to a partitioned area to which a second coefficient is applied, the coefficient of the mask processing applied to the same protection object is smoothly changed from the first coefficient to the second coefficient in a plurality of subsequent images. The image processing device according to claim 1 .

6. Detecting a target area in which an object to be protected for personal information is captured from an image output by an imaging device mounted on the vehicle; Identifying a driving scene according to the driving lane and vehicle speed of the host vehicle; determining a coefficient representing the strength of mask processing for each divided area of the image in accordance with the driving scene; A mask process is performed on the target region according to the coefficient. An image processing method in which processing is performed by an in-vehicle device.

7. Detecting a target area in which an object to be protected for personal information is captured from an image output by an imaging device mounted on the vehicle; Identifying a driving scene according to the driving lane and vehicle speed of the host vehicle; determining a coefficient representing the strength of mask processing for each divided area of the image in accordance with the driving scene; A mask process is performed on the target region according to the coefficient. An image processing program for causing an in-vehicle device to execute the processing.

Images