Computing device and privacy protection method

Abstract

To generate in-car footage that is both aesthetically pleasing and appealing as video content, while also respecting privacy. [Solution] The computing device according to the embodiment includes a controller. The controller acquires in-vehicle video footage captured by an in-vehicle camera. The controller also acquires vehicle location information. The controller also acquires location information of an object to be concealed. Based on the vehicle location information and the location information of the object to be concealed, the controller determines whether the object to be concealed falls within the field of view of the in-vehicle camera corresponding to each of the multiple window areas in the in-vehicle video footage. If the controller determines that the object to be concealed falls within any of the fields of view, it executes a concealment process to conceal the window area corresponding to the field of view.

Description

Technical Field

[0001] The disclosed embodiments relate to an arithmetic unit and a privacy protection method.

Background Art

[0002] Conventionally, in order to prevent personal information outside the vehicle from being reflected in an in-vehicle video obtained by photographing the interior of a vehicle, a technique of masking an area of a window portion in the in-vehicle video (hereinafter, appropriately referred to as a “window area”) is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in recent years, with the spread of video distribution services and the like, the needs of users who want to use in-vehicle videos captured by vehicles as video content have been increasing. However, when the above-described conventional technology is used, there is a problem that the window area is uniformly masked, and the attractiveness as video content is reduced.

[0005] One aspect of the embodiment has been made in view of the above, and an object is to provide an arithmetic unit and a privacy protection method capable of generating an in-vehicle video that is attractive as video content while considering privacy protection.

Means for Solving the Problems

[0006] A computing device according to one embodiment includes a controller. The controller acquires in-vehicle video footage captured by an in-vehicle camera. The controller also acquires vehicle location information. The controller also acquires location information of an object to be concealed. Based on the vehicle location information and the location information of the object to be concealed, the controller determines whether the object to be concealed falls within the field of view of the in-vehicle camera corresponding to each of a plurality of window areas in the in-vehicle video footage. If the controller determines that the object to be concealed falls within any of the fields of view, it executes a concealment process to conceal the window area corresponding to the field of view. [Effects of the Invention]

[0007] According to one embodiment, the controller in the computing device determines, based on the vehicle's location information and the location information of the object to be concealed, whether the object to be concealed falls within the field of view corresponding to each of the multiple window areas in the in-car video. The controller then conceals only the window areas corresponding to the field of view in which the object to be concealed is determined to be included. This allows the controller to prevent the entire window area from being uniformly concealed in the in-car video while preserving the appeal of the video content. In other words, according to one embodiment, it is possible to generate in-car video that is both privacy-protecting and appealing as video content. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is an overview diagram (part 1) of the privacy protection method according to the embodiment. [Figure 2] Figure 2 is an overview diagram (part 2) of the privacy protection method according to the embodiment. [Figure 3] Figure 3 is an overview diagram (part 3) of the privacy protection method according to the embodiment. [Figure 4] Figure 4 is an overview diagram (part 4) of the privacy protection method according to the embodiment. [Figure 5]Figure 5 shows an example of the configuration of an in-vehicle device according to this embodiment. [Figure 6] Figure 6 is a flowchart showing the processing procedure performed by the in-vehicle device according to the embodiment. [Figure 7] Figure 7 is a diagram (part 1) illustrating an example of a case where an object detected by an in-vehicle sensor is to be concealed. [Figure 8] Figure 8 is a diagram (part 2) illustrating an example of a case where an object detected by an in-vehicle sensor is to be concealed. [Figure 9] Figure 9 is a diagram (part 3) illustrating an example of a case where an object detected by an in-vehicle sensor is to be concealed. [Figure 10] Figure 10 is a diagram (part 1) illustrating an example of concealment using dummy footage. [Figure 11] Figure 11 is a diagram (part 2) illustrating an example of concealment using dummy footage. [Figure 12] Figure 12 shows an example of the configuration of a privacy protection system according to an embodiment. [Figure 13] Figure 13 shows an example of the configuration of a center device according to an embodiment. [Figure 14] Figure 14 shows a modified example of a memory unit. [Figure 15] Figure 15 is an explanatory diagram (part 1) of a modified example using generative AI. [Figure 16] Figure 16 is an explanatory diagram (part 2) of a modified example using generative AI. [Modes for carrying out the invention]

[0009] The embodiments of the computing device and privacy protection method disclosed herein will be described in detail below with reference to the attached drawings. However, the present invention is not limited to the embodiments described below.

[0010] Furthermore, in the following, the computing device according to the embodiment shall be either an in-vehicle device 10 (see Figure 5) or a center device 100 (see Figure 13). The privacy protection method according to the embodiment shall be a privacy protection method executed by the controller 14 (see Figure 5) of the in-vehicle device 10 or the controller 104 (see Figure 13) of the center device 100.

[0011] Furthermore, the expressions "specific," "designated," and "certain" in the following explanation may be interpreted as "predetermined."

[0012] First, an overview of the privacy protection method according to the embodiment will be explained using Figures 1 and 2. Figure 1 is an overview diagram (part 1) of the privacy protection method according to the embodiment. Figure 2 is an overview diagram (part 2) of the privacy protection method according to the embodiment.

[0013] Figure 3 is a schematic diagram (part 3) illustrating the privacy protection method according to the embodiment. Figure 4 is a schematic diagram (part 4) illustrating the privacy protection method according to the embodiment. In the explanation using Figures 1 to 11, the case in which the in-vehicle device 10 is an example of a computing device is given as an example.

[0014] In the privacy protection method according to this embodiment, the controller 14 of the in-vehicle device 10 acquires in-vehicle video footage captured by the in-vehicle camera. The controller 14 also acquires the location information of the vehicle. The controller 14 also acquires the location information of the object to be concealed. Based on the location information of the vehicle and the location information of the object to be concealed, the controller 14 determines whether the object to be concealed falls within the field of view of the in-vehicle camera corresponding to each of the multiple window areas in the in-vehicle video footage. If the controller 14 determines that the object to be concealed falls within any of the above fields of view, it executes a concealment process to conceal the window area corresponding to the field of view.

[0015] This will be explained in more detail using Figures 1 and 2. Figure 1 schematically shows the in-vehicle image Mi before the privacy protection method according to the embodiment is applied in the upper figure, and the in-vehicle image Mi after the same privacy protection method is applied in the lower figure. The in-vehicle image Mi corresponds to one frame of camera video data that captures the in-vehicle image.

[0016] The in-vehicle device 10 is implemented, for example, as an image processing device or a drive recorder mounted in the vehicle. As shown in Figure 1, in the privacy protection method according to this embodiment, the controller 14 acquires an in-vehicle image Mi captured by the in-vehicle camera 5a (see Figure 2). Each occupant C that appears in the in-vehicle image Mi corresponds to an example of a "user". The controller 14 then performs region classification processing on the in-vehicle image Mi, such as semantic segmentation, extracts multiple window regions, and labels each window region.

[0017] As shown in Figure 1, the controller 14 designates the window area corresponding to the vehicle's rear window as window area B. The controller 14 also designates the window areas corresponding to the vehicle's left side window as window areas L1 and L2, starting from the front of the passenger compartment. The controller 14 also designates the window areas corresponding to the vehicle's right side window as window areas R1 and R2, starting from the front of the passenger compartment.

[0018] The controller 14 then determines, based on the vehicle's location information corresponding to the shooting position of the in-vehicle image Mi and the location information of the object to be concealed, whether the object to be concealed is within the field of view of the in-vehicle camera 5a corresponding to the window areas B, L1, L2, R1, and R2 in the in-vehicle image Mi.

[0019] As shown in Figure 2, the in-vehicle camera 5a is installed to capture images of the interior of the vehicle from the front of the vehicle compartment and has a field of view θ. And, through the rear window, which is one of the windows separated by each pillar, the field of view θ is included in the field of view θ. B Objects outside the vehicle and the background inside are captured by the in-car camera 5a. In other words, the window area B has a field of view θ. BExternal objects and the background outside the vehicle will be reflected therein.

[0020] Similarly, in the window regions L1 and L2 corresponding to the left side window, angular fields θ L1 , θ L2 External objects and the background outside the vehicle will be reflected therein. Similarly, in the window regions R1 and R2 corresponding to the right side window, angular fields θ R1 , θ R2 External objects and the background outside the vehicle will be reflected therein.

[0021] Therefore, based on the position information of the vehicle and the position information of the object to be anonymized, the controller 14 determines whether the object to be anonymized enters the angular fields θ B , θ L1 , θ L2 , θ R1 , θ R2 corresponding to the window regions B, L1, L2, R1, and R2 respectively. Note that "the object to be anonymized enters the angular field" includes not only the case where the object to be anonymized is currently within the angular field but also, for example, the case where it is predicted that the object to be anonymized will enter the angular field within a predetermined time.

[0022] When the controller 14 determines that the object to be anonymized enters any of these angular fields, it turns on the anonymization function to perform anonymization processing such as masking individually for the corresponding window region. Also, the controller 14 turns off the anonymization function so as not to perform anonymization processing individually for the non-corresponding window region.

[0023] Specifically, as shown in FIG. 3, the controller 14, for example, turns off the anonymization function for all window regions. Then, while moving in the direction of arrow a1 in that state, the controller 14 determines whether a facility where photography is prohibited enters any of the angular fields based on the position information of the vehicle and the position information of the facility where photography is prohibited. The position information of the facility where photography is prohibited is obtained, for example, from map information or the like.

[0024] And the controller 14 determines the angular fields θ B , θ L1 , θ L2,θ R1 ,θ R2 If any of the areas within the frame contain a facility where photography is prohibited, the corresponding field of view θ B ,θ L1 ,θ L2 ,θ R1 ,θ R2 The concealment function is turned on for the corresponding window areas B, L1, L2, R1, and R2.

[0025] Figure 3 shows an example where the controller 14 turns on the concealment function for window areas B, R1, and R2 while the vehicle is passing to the left of a facility where photography is prohibited. The controller 14 repeatedly turns this concealment function on and off until the vehicle has passed the facility where photography is prohibited and the facility is no longer included in the field of view θ of the in-vehicle camera 5a.

[0026] Furthermore, as shown in Figure 4, the controller 14 keeps, for example, the concealment function turned on for each window area. Then, while moving in the direction of arrow a2 in this state, the controller 14 determines whether the shooting area falls within any of the fields of view, based on the vehicle's position information and the position information of the shooting area. The position information of the shooting area is obtained from, for example, map information or arbitrary setting information.

[0027] Then, the controller 14 controls the field of view θ B ,θ L1 ,θ L2 ,θ R1 ,θ R2 If the shooting area falls within any of the above, the corresponding field of view θ B ,θ L1 ,θ L2 ,θ R1 ,θ R2 The concealment function will be turned off for the corresponding window areas B, L1, L2, R1, and R2.

[0028] Figure 4 shows an example where the controller 14 turns off the concealment function for window areas B, R1, and R2 while the vehicle is passing to the right of the area that can be photographed. The controller 14 repeatedly turns this concealment function on and off until the vehicle passes alongside the area that can be photographed and the area is no longer included in the field of view θ of the in-vehicle camera 5a.

[0029] Let's return to the explanation of Figure 1. Figure 1 shows an example where, due to this on / off control of the concealment function, the in-car image Mi in the upper figure had the concealment function turned off for all window areas, while in the lower figure the concealment function was turned on for window areas B, L1, and L2, and they were concealed, for example, by masking.

[0030] Furthermore, the controller 14 may also conceal not only the no-photography facilities shown in Figure 3 (corresponding to an example of a "no-photography area or facility"), but also other vehicles, people, or any building set by the user (see window areas B, L1, and L2 in the figure, respectively).

[0031] Furthermore, the controller 14 may also choose to conceal a predetermined range based on the user's home or workplace, as set by the user. In this case, the controller 14 turns on the concealment function for all window areas B, L1, L2, R1, and R2 within that range. This prevents the user's home or workplace from being identified by a third party.

[0032] Furthermore, the controller 14 may also conceal objects detected by in-vehicle sensors, including in-vehicle cameras and radar. Other examples of such concealment will be described later with reference to Figures 7 to 9.

[0033] As described above, in the privacy protection method according to this embodiment, the controller 14 of the in-vehicle device 10 acquires in-vehicle video footage captured by the in-vehicle camera 5a (corresponding to an example of an "in-vehicle camera"). The controller 14 also acquires the location information of the vehicle. The controller 14 also acquires the location information of the object to be concealed. Based on the vehicle's location information and the location information of the object to be concealed, the controller 14 determines the field of view θ of the in-vehicle camera 5a corresponding to the window regions B, L1, L2, R1, R2 in the in-vehicle video footage. B ,θ L1 ,θ L2 ,θ R1 ,θ R2 The controller determines whether the object to be concealed is included. Additionally, the controller 14 determines the field of view θ. B ,θ L1 ,θ L2 ,θ R1 ,θ R2 If it is determined that the object to be concealed is located within any of the fields, a concealment process is executed to conceal the window area corresponding to that field of view.

[0034] Therefore, according to the privacy protection method according to the embodiment, the controller 14 determines the field of view θ corresponding to the window regions B, L1, L2, R1, R2 in the in-vehicle video, based on the vehicle's location information and the location information of the object to be concealed. B ,θ L1 ,θ L2 ,θ R1 ,θ R2 The controller 14 determines whether the object to be concealed is included in the field of view. Then, the controller 14 conceals only the window area corresponding to the field of view in which the object to be concealed is included. In this way, the controller 14 can prevent the window areas B, L1, L2, R1, and R2 from being uniformly concealed in the in-car video while preserving the appeal of the video content. In other words, according to one embodiment, it is possible to generate in-car video that is appealing as video content while taking privacy protection into consideration.

[0035] The following describes in more detail an example of the configuration of the in-vehicle device 10 to which the privacy protection method according to the above embodiment is applied.

[0036] Figure 5 shows an example of the configuration of the in-vehicle device 10 according to the embodiment. As already mentioned, the in-vehicle device 10 is implemented as an image processing device or a drive recorder mounted on a vehicle.

[0037] As shown in Figure 5, the in-vehicle device 10 includes a communication unit 11, an HMI unit 12, a storage unit 13, and a controller 14. The in-vehicle device 10 is also connected to an in-vehicle sensor 5.

[0038] The in-vehicle sensor 5 is a group of various sensors mounted on the vehicle. The in-vehicle sensor unit is connected to the in-vehicle device 10 via an in-vehicle network such as CAN (Controller Area Network).

[0039] The in-vehicle sensor 5 includes an in-vehicle camera 5a, an out-of-vehicle camera 5b, a GPS (Global Positioning System) sensor 5c, a vehicle speed sensor 5d, and a radar 5e. The in-vehicle camera 5a and the out-of-vehicle camera 5b are examples of "in-vehicle cameras".

[0040] The interior camera 5a is installed to capture images of the interior of the vehicle from the front of the vehicle compartment. The interior camera 5a is mounted near the windshield or dashboard, etc. The exterior camera 5b includes a front camera 5b-F (see Figure 7), a rear camera 5b-R (see Figure 8), and a side camera (not shown).

[0041] The front camera 5b-F is positioned to capture the area in front of the vehicle. The rear camera 5b-R is positioned to capture the area behind the vehicle. The side cameras are positioned to capture the area to the sides of the vehicle.

[0042] The GPS sensor 5c determines the vehicle's GPS position and outputs it as position information. The vehicle speed sensor 5d outputs a vehicle speed pulse signal as vehicle speed information. The radar 5e detects objects present around the vehicle using millimeter waves.

[0043] In addition, the in-vehicle sensor 5 may include various other sensors besides those shown in Figure 5. For example, the in-vehicle sensor 5 may include a G-sensor or LiDAR (Light Detection and Ranging). Furthermore, some of the sensors included in the in-vehicle sensor 5 may be mounted on the in-vehicle device 10.

[0044] The communication unit 11 is implemented by a network adapter or the like. The communication unit 11 is connected to other in-vehicle devices via an in-vehicle network such as CAN, and transmits and receives information with other in-vehicle devices.

[0045] Furthermore, the communication unit 11 is wirelessly connected to the center device 100, which will be described later, via a network such as the Internet, a mobile phone network, or a C-V2X (Cellular Vehicle to Everything) communication network. The communication unit 11 transmits and receives information with the center device 100 via this network.

[0046] The HMI unit 12 is a component that provides interface components for input and output to the user operating the in-vehicle device 10. The HMI unit 12 includes an input interface that accepts input operations from the user. The input interface is implemented, for example, by a touch panel. Alternatively, the input interface may be implemented by a microphone or the like. Furthermore, the input interface may be implemented by software components.

[0047] Furthermore, the HMI unit 12 includes an output interface for presenting visual and auditory information to the user. The output interface is implemented, for example, by a display or speaker. The HMI unit 12 may also provide the input interface and output interface to the user as an integrated unit, for example, by a touch panel display.

[0048] The storage unit 13 is implemented by storage devices such as ROM (Read Only Memory), RAM (Random Access Memory), and flash memory. The storage unit 13 also includes a ring buffer memory. In the example shown in Figure 5, the storage unit 13 stores map information DB (Database) 13a, setting information 13b, camera image data DB 13c, image recognition model 13d, anonymization processing information 13e, and anonymized data DB 13f.

[0049] Map Information DB13a is a database of map information. The map information stored in Map Information DB13a is linked to location information of items to be anonymized. The items to be anonymized included in the map information are at least areas or facilities where photography is prohibited. This allows for the sharing of predetermined common items to be anonymized based on map information.

[0050] Configuration information 13b includes various settings related to the concealment process. Configuration information 13b includes information about the data to be concealed, which is arbitrarily set by the user, for example, via the HMI unit 12. The location information of the data to be concealed included in configuration information 13b is appropriately linked to the map information included in the map information DB 13a.

[0051] The camera video data DB13c is a database of camera video data captured by the in-vehicle camera 5a and the exterior camera 5b. The camera video data is associated with the vehicle's position information at the time of capture. The camera video data DB13c is recorded, for example, in a ring buffer memory.

[0052] Image recognition model 13d is an AI (Artificial Intelligence) model for image recognition. The AI ​​model is a Deep Neural Network (DNN) model trained using machine learning algorithms, for example. This AI model is pre-trained to detect the type, location, color, etc., of various objects that appear in video data captured by an in-vehicle camera. The AI ​​model is used, for example, to classify window areas in in-vehicle video and to detect objects to be concealed in exterior video.

[0053] After the image recognition model 13d is loaded into the controller 14 as an AI model, it operates as an image recognition AI when each frame of video data captured by the in-vehicle camera 5a and the exterior camera 5b is input to the controller 14.

[0054] The concealment processing information 13e is information containing various parameters used when the concealment processing is performed. For example, the concealment processing information 13e includes information indicating whether the aforementioned concealment function corresponding to each window area B, L1, L2, R1, R2 is turned on or off.

[0055] The anonymized data DB13f is a database that stores camera video data, including anonymized data of in-vehicle video output after anonymization processing. The camera video data stored in the anonymized data DB13f can be output to the HMI unit 12. The camera video data can also be transmitted to the center device 100 via the communication unit 11. Furthermore, the camera video data can be stored in an external device that enables viewing of video content, etc., via the communication unit 11 or a recording medium, etc.

[0056] The controller 14 corresponds to a so-called processor. The controller 14 is implemented by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), etc. The controller 14 executes a program according to an embodiment not shown, stored in the memory unit 13, using RAM as the working area. The controller 14 can also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0057] The controller 14 performs information processing according to the processing procedure shown in Figure 6. Next, this processing procedure will be described. Figure 6 is a flowchart of the processing procedure performed by the in-vehicle device 10 according to the embodiment. Note that the processing procedure in Figure 6 shows the processing procedure for in-vehicle image Mi, which is one frame of camera video data that captures in-vehicle video. Therefore, the processing procedure in Figure 6 is repeated for each frame of the in-vehicle video.

[0058] As shown in Figure 6, first the controller 14 acquires an in-vehicle image Mi from the camera video data stored in the camera video data DB 13c (step S101). The controller 14 also acquires the vehicle's location information corresponding to the in-vehicle image Mi (step S102).

[0059] Furthermore, the controller 14 uses the map information DB 13a to acquire location information to be concealed that corresponds to the vicinity of the vehicle's location information (step S103). The vicinity of the vehicle's location information is, for example, a predetermined range based on the vehicle's location information.

[0060] Then, the controller 14 determines, based on the acquired vehicle location information and the location information of the object to be concealed, whether or not there is a window area in the in-vehicle image Mi that contains the object to be concealed within the corresponding field of view (step S104).

[0061] If a corresponding window area exists (step S104, Yes), the controller 14 turns on the window area concealment function (not shown) and conceals the corresponding window area (step S105). The controller 14 conceals the window area by, for example, masking. Alternatively, the controller 14 may conceal the window area by, for example, mosaic processing. Or, the controller 14 may conceal the window area by, for example, drawing a dummy image. Examples of dummy images will be described later using Figures 10 and 11.

[0062] Next, the controller 14 combines the concealed window area with the in-car image Mi (step S106). Then, the controller 14 outputs the concealed in-car image Mi to the concealed data DB13f (step S107), and terminates the process.

[0063] Furthermore, if there is no corresponding window area (step S104, No), the controller 14 turns off the concealment function for all window areas (not shown), outputs the in-vehicle image Mi acquired in step S101 to the concealed data DB13f (step S107), and terminates the process.

[0064] In the processing procedure shown in Figure 6, the location information of the object to be concealed is obtained using map information in which the object to be concealed has been set. However, the controller 14 may also obtain the location information of an object detected by the on-board sensor 5 from the object detection result.

[0065] Figure 7 is a diagram (part 1) showing an example of a case where an object detected by the in-vehicle sensor 5 is to be concealed. Figure 8 is a diagram (part 2) showing an example of a case where an object detected by the in-vehicle sensor 5 is to be concealed. Figure 9 is a diagram (part 3) showing an example of a case where an object detected by the in-vehicle sensor 5 is to be concealed.

[0066] As shown in Figure 7, while moving in the direction of arrow a3, the controller 14 performs image recognition processing using the image recognition model 13d on the front camera video data captured by, for example, the front cameras 5b-F.

[0067] Assuming that this image recognition process detects a pedestrian W on the front left side of the vehicle, as shown in Figure 7, the controller 14 then identifies the pedestrian W as a target to be concealed. The controller 14 then obtains the location information of the pedestrian W from the detection result and determines whether the pedestrian W is within the field of view corresponding to each window area of ​​the in-vehicle camera 5a, based on the vehicle's location information and the pedestrian W's location information.

[0068] At this time, even if the pedestrian W is not currently in the field of view, the controller 14 predicts, for example, the time until the pedestrian W enters the field of view in the direction of travel indicated by arrow a3, based on vehicle speed information. If the predicted time is within a predetermined time, the controller 14 turns on the concealment function for window areas B, L1, and L2, as in the example in Figure 7. This allows for concealment of the in-vehicle video footage of the object to be concealed detected in front of the vehicle, based on the detection results of the in-vehicle sensor 5 and the predictions made from those detection results.

[0069] Furthermore, even if pedestrian W is not currently within the field of view, if it can be determined that the vehicle will definitely pass beside pedestrian W in the direction of travel, the concealment function may be immediately turned on.

[0070] Furthermore, as shown in Figure 8, while moving in the direction of arrow a4, the controller 14 performs image recognition processing using the image recognition model 13d on the rear camera video data captured by, for example, the rear camera 5b-R.

[0071] As shown in Figure 8, this image recognition process detects a following vehicle (RV) behind the vehicle. In this case, the controller 14 identifies the following vehicle (RV) as a target for concealment. The controller 14 then obtains the location information of the following vehicle (RV) from the detection result and, based on the vehicle's location information and the following vehicle (RV)'s location information, determines the field of view θ corresponding to the window area B of the in-vehicle camera 5a. B Determine if a following RV vehicle will enter the space.

[0072] Then, the controller 14 controls the field of view θ B When it is determined that a following vehicle (RV) is entering the area, the concealment function for window area B is turned on. The controller 14 controls the field of view θ. B Even if a following RV is present, if the distance between vehicles is large enough that, for example, the license plate or occupants of the following RV cannot be identified, it is not necessary to turn on the window area B concealment function. This prevents the window area from being unnecessarily concealed, which would reduce the appeal of the video content.

[0073] Figures 7 and 8 illustrate an example of concealing people and other vehicles detected by the on-board sensor 5, but other elements such as route guidance signs and location guidance signs could also be concealed. This prevents locations from being identified from place names and other information displayed on each sign.

[0074] Furthermore, the controller 14 detects objects that change their relative position to its own vehicle, for example, using radar 5e. Through this detection process, as shown in Figure 9, the controller 14 detects a following vehicle RV that is approaching (see arrow a5) while traveling in the lane to the right of its own lane using radar 5e.

[0075] In this case, the controller 14 identifies the following vehicle RV as a target for concealment. The controller 14 then obtains the location information of the following vehicle RV from the detection result of the following vehicle RV, and determines whether the following vehicle RV is within the field of view corresponding to the window area of ​​the in-vehicle camera 5a based on the vehicle's location information and the following vehicle RV's location information.

[0076] Then, if the controller 14 determines that a following vehicle RV is within the field of view, it turns on the concealment function for the corresponding window area. As shown in the upper diagram of Figure 9, for example, if a following vehicle RV is located diagonally to the right and behind the vehicle, and the radar 5e located near the rear left side detects this, the controller 14 turns on the concealment function for window areas B and R2.

[0077] Furthermore, as shown in the lower diagram of Figure 9, if, for example, a following vehicle RV is passing to the right of the vehicle and the radar 5e located near the left side detects this, the controller 14 will turn off the concealment function for window area B and turn on the concealment functions for window areas R1 and R2.

[0078] This allows for the concealment of targets such as other vehicles passing alongside the vehicle, by switching the window area according to changes in their position.

[0079] Up to this point, we have mainly shown examples of concealing each window area by masking, but concealment can also be performed by drawing a dummy image in the window area, for example. Figure 10 is a diagram (part 1) showing an example of concealment using a dummy image. Figure 11 is a diagram (part 2) showing an example of concealment using a dummy image.

[0080] The masking methods described so far mostly involve filling the window area with a single-color pattern. In contrast, the controller 14 may conceal the window area using a dummy image instead of a simple pattern like masking. Figures 10 and 11 show examples of dummy images corresponding to window area B.

[0081] As shown in Figure 10, the dummy video may depict a virtual object such as an animal or character (in this case, a cat). Furthermore, when used as video content distributed on video streaming services, it may display advertisements, as shown in Figure 10.

[0082] Furthermore, the dummy video may be, for example, a video of the virtual object or advertisement mentioned above. In this case, the controller 14 may adjust the playback speed of the dummy video according to, for example, the vehicle speed information from the vehicle speed sensor 5d.

[0083] Alternatively, the dummy image may be modified in conjunction with the vehicle's movement (e.g., vehicle speed information and direction of travel information) by determining the direction and amount of movement of each pixel within each window area using image analysis processing such as optical flow.

[0084] For example, the upper diagram in Figure 11 schematically represents the optical flow that flows in a concentrated manner towards the vanishing point at the rear of the vehicle within the window area B. Based on this optical flow, the controller 14 may play a dummy image, such as a shooting star, depending on the direction and amount of movement indicated by the optical flow, as shown in the lower diagram in Figure 11. Note that Figures 10 and 11 are merely examples and do not limit the form of the dummy image.

[0085] By using dummy footage in this way to conceal the original video, it becomes possible to generate engaging and non-monotonous in-car footage while taking privacy protection into consideration.

[0086] Furthermore, while the above description has focused on the case where the in-vehicle device 10 is an example of a computing device according to the embodiment, the center device 100 may similarly be an example of a computing device. Next, we will describe this case.

[0087] Figure 12 shows an example of the configuration of the privacy protection system 1 according to an embodiment. As shown in Figure 12, the privacy protection system 1 includes in-vehicle devices 10-1, 10-2, ..., 10-m (where m is a natural number of 3 or greater) and a center device 100.

[0088] Each in-vehicle device 10 and the central device 100 are connected to each other via a network N1, such as the Internet, a mobile phone network, or a C-V2X communication network, enabling them to communicate with one another.

[0089] Since the in-vehicle device 10 has already been explained, a detailed explanation will be omitted here. Note that when the central device 100 is a computing device according to the embodiment, the in-vehicle device 10 does not perform the anonymization process described above, but instead transmits camera video data, including in-vehicle video, and vehicle location information corresponding to the shooting location to the central device 100.

[0090] The central device 100 is implemented, for example, as a private cloud. The central device 100 is managed, for example, by a company that operates the data center for the in-vehicle devices 10. The central device 100 collects camera video data and vehicle location information transmitted from each in-vehicle device 10.

[0091] Next, an example of the configuration of the center device 100 will be described. Figure 13 is a diagram showing an example of the configuration of the center device 100 according to the embodiment. As shown in Figure 13, the center device 100 includes a communication unit 101, an HMI unit 102, a storage unit 103, and a controller 104.

[0092] The communication unit 101 is implemented by a network adapter or the like. The communication unit 101 is connected to the network N1 by wire or wireless connection and transmits and receives information to and from each in-vehicle device 10 via the network N1.

[0093] The HMI unit 102 is a component that provides interface components for input and output to operators, etc., who operate the center device 100. The HMI unit 102 includes an input interface that receives input operations from operators, etc. The input interface is implemented, for example, by a touch panel. Alternatively, the input interface may be implemented by a keyboard, mouse, pen tablet, microphone, etc. Furthermore, the input interface may be implemented by software components.

[0094] Furthermore, the HMI unit 102 includes an output interface for presenting visual and audio information to the operator. The output interface is implemented, for example, by a display or speaker. Alternatively, the HMI unit 102 may provide the input interface and output interface to the operator as an integrated unit, for example, by using a touch panel display.

[0095] The storage unit 103 is implemented by a storage device such as ROM, RAM, flash memory, or HDD (Hard Disk Drive). In the example shown in Figure 13, the storage unit 103 stores map information DB 103a, setting information 103b, camera image data DB 103c, image recognition model 103d, anonymization processing information 103e, and anonymized data DB 103f.

[0096] Map information DB103a corresponds to the map information DB13a described above. Configuration information 103b corresponds to the configuration information 13b described above. Camera video data DB103c corresponds to the camera video data DB13c described above. Image recognition model 103d corresponds to the image recognition model 13d described above. Anonymization processing information 103e corresponds to the anonymization processing information 13e described above. Anonymized data DB103f corresponds to the anonymized data DB13f described above.

[0097] These are all similar, but in the central device 100, at least the map information DB 103a, setting information 103b, camera image data DB 103c, and concealed data DB 103f have identification information of each in-vehicle device 10 associated with each of the stored data.

[0098] The controller 104 corresponds to a so-called processor. The controller 104 is implemented by a CPU, MPU, GPU, etc. The controller 104 executes a program according to an embodiment not shown, stored in the memory unit 103, using RAM as the working area. The controller 104 can also be implemented by an integrated circuit such as an ASIC or FPGA.

[0099] Controller 104, like Controller 14 described above, performs information processing according to the processing procedure shown in Figure 6. As a result, Controller 104 stores the camera video data, including the encrypted data of the in-vehicle video output after at least the encryption process, in the encrypted data DB 103f. The camera video data stored in the encrypted data DB 103f can be output to the HMI unit 102. The camera video data can also be transmitted to each in-vehicle device 10 via the communication unit 11. Controller 104 may also provide a video distribution service that distributes the camera video data as video content.

[0100] Next, we will describe a modified example using a generation AI. This modified example can be performed whether the computing device is the in-vehicle device 10 or the center device 100. Here, we will use the in-vehicle device 10 as an example.

[0101] Figure 14 shows a modified memory unit 13A. Figure 15 is an explanatory diagram (part 1) of a modified example using generation AI. Figure 16 is an explanatory diagram (part 2) of a modified example using generation AI.

[0102] As shown in Figure 14, the modified memory unit 13A differs from the memory unit 13 in the embodiment in that it further stores the generated AI model 13g (see Figure 5). The generated AI model 13g is an AI model for generating estimated images for each window region. This AI model is pre-trained to be able to generate estimated images corresponding to each window region when front camera image data, rear camera image data, and vehicle speed information are input.

[0103] The trained generative AI model 13g is loaded into the controller 14 as an AI model, and then operates as a generative AI when front camera video data, rear camera video data, and vehicle speed information are input to the controller 14.

[0104] Specifically, as shown in Figure 15, the generating AI model 13g is pre-trained using training data that takes front camera video data, rear camera video data, and vehicle speed information as input data, and uses the video of each window area of ​​the in-car camera as ground truth data.

[0105] Furthermore, as shown in Figure 16, the trained generative AI model 13g is loaded into the controller 14 and operates as a generative AI. When this generative AI receives real-time data during driving, such as front camera video data, rear camera video data, and vehicle speed information, it outputs estimated images of each window area of ​​the in-vehicle camera 5a.

[0106] When the controller 14 acquires this estimated video, it performs region classification and object detection on the estimated video using an AI model, etc., and extracts regions to be concealed, such as people and other vehicles. The controller 14 then performs a concealment removal process to erase the extracted regions to be concealed, and obtains video of each window region after the concealment has been removed. The controller 14 then composites the video of each window region after processing into the window region of the in-car video.

[0107] Although not shown in the diagram, as shown in Figure 16, instead of removing the areas to be concealed from the estimated image as a post-processing step, it is also possible to perform area classification or object detection using an AI model or the like as a pre-processing step on the ground truth data, and remove the areas to be concealed from the ground truth data in advance.

[0108] By modifying the data using this generation AI, high-quality video footage of each window area, with the hidden elements removed, can be easily obtained and composited into the in-car video. In other words, it is possible to generate in-car video that is attractive as video content while taking privacy protection into consideration.

[0109] As described above, the in-vehicle device 10 according to the embodiment (corresponding to an example of a "processing unit") includes a controller 14. The controller 14 acquires in-vehicle video footage captured by the in-vehicle camera 5a (corresponding to an example of an "in-vehicle camera"). The controller 14 also acquires vehicle location information. The controller 14 also acquires location information of the object to be concealed. Based on the vehicle location information and the location information of the object to be concealed, the controller 14 determines the field of view θ of the in-vehicle camera 5a corresponding to the window regions B, L1, L2, R1, R2 in the in-vehicle video footage. B ,θ L1 ,θ L2 ,θ R1 ,θ R2 The controller determines whether the object to be concealed is included. Additionally, the controller 14 determines the field of view θ. B ,θ L1 ,θ L2 ,θ R1 ,θ R2 If it is determined that the object to be concealed is located within any of the fields, a concealment process is executed to conceal the window area corresponding to that field of view.

[0110] Therefore, according to the in-vehicle device 10 of this embodiment, the controller 14 determines the field of view θ corresponding to the window regions B, L1, L2, R1, R2 in the in-vehicle video, based on the vehicle's location information and the location information of the object to be concealed. B ,θ L1 ,θ L2 ,θ R1 ,θ R2 The controller 14 determines whether the object to be concealed is included in the field of view. Then, the controller 14 conceals only the window area corresponding to the field of view in which the object to be concealed is included. In this way, the controller 14 can prevent the window areas B, L1, L2, R1, and R2 from being uniformly concealed in the in-car video while preserving the appeal of the video content. In other words, according to one embodiment, it is possible to generate in-car video that is appealing as video content while taking privacy protection into consideration.

[0111] Further effects and modifications can be readily derived by those skilled in the art. Therefore, broader aspects of the present invention are not limited to the specific details and representative embodiments expressed and described above. Accordingly, various modifications are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and their equivalents. [Explanation of Symbols]

[0112] 1. Privacy protection system 5. In-vehicle sensors 10 Onboard equipment 11 Communications Department 12 HMI section 13 Storage section 14 Controllers 100 Center device 101 Communications Department 102 HMI section 103 Storage section 104 Controller

Claims

1. Equipped with a controller, The aforementioned controller, We acquire in-car video footage captured by the in-car camera, Obtain vehicle location information, Obtain location information of items to be kept secret, Based on the location information of the vehicle and the location information of the object to be concealed, it is determined whether the object to be concealed falls within the field of view of the in-vehicle camera corresponding to each of the multiple window areas in the in-vehicle video. If it is determined that the object to be concealed falls within any of the aforementioned fields of view, a concealment process is executed to conceal the window area corresponding to the relevant field of view. Computing device.

2. The aforementioned controller, The object to be concealed acquires location information of the object based on pre-configured map information. The computing device according to claim 1.

3. The subject of the secrecy is an area or facility where photography is prohibited. The arithmetic device according to claim 2.

4. The aforementioned controller, The scope of the confidentiality to be determined by the user of the vehicle, based on the user's home or workplace. The computing device according to claim 1.

5. The controller designates objects detected by the vehicle-mounted sensors as targets for concealment. The computing device according to claim 1.

6. The aforementioned object is a person, another vehicle, a route guidance sign, or a point guidance sign. The arithmetic device according to claim 5.

7. The controller conceals the window area by drawing a pre-generated dummy image onto the window area. The computing device according to claim 1.

8. The aforementioned controller, The vehicle speed information and direction of travel information of the aforementioned vehicle are acquired. The rendering of the dummy image is changed according to the vehicle speed information and the direction of travel information. The arithmetic device according to claim 7.

9. Equipped with a controller, The aforementioned controller, The vehicle acquires external and internal video footage captured by the in-vehicle camera. Obtain vehicle speed information, When the external video footage and the vehicle speed information are input, a generation AI trained to generate estimated video footage corresponding to multiple window regions in the interior video footage is used to perform an anonymization process in which the estimated video footage, from which the anonymized elements to be anonymized have been removed, is synthesized with the interior video footage. Computing device.

10. A privacy protection method performed by the controller, This involves acquiring in-car video footage captured by an in-car camera, and Obtaining vehicle location information, Obtaining location information of items that should be kept confidential, Based on the location information of the vehicle and the location information of the object to be concealed, it is determined whether the object to be concealed falls within the field of view of the in-vehicle camera corresponding to each of the multiple window areas in the in-vehicle video, If it is determined that the object to be concealed falls within any of the aforementioned fields of view, a concealment process is performed to conceal the window area corresponding to the relevant field of view. Privacy protection methods including

Images