Anonymizing personally identifiable information in image data

Abstract

The present disclosure provides "anonymizing personally identifiable information in image data". A computer includes a processor and a memory, and the memory stores instructions executable by the processor to identify personally identifiable information (PII) in an initial image frame; encoding a region of the initial image frame containing the PII as a potential vector having a lower dimension than the region; anonymizing the region from the initial image frame, thereby producing a modified image frame; and associating the potential vector with the modified image frame.

Description

Technical Field

[0001] The present disclosure relates to anonymization of personally identifiable information in image data. Background Art

[0002] A vehicle may include a variety of sensors. Some sensors detect the internal state of the vehicle, such as wheel speed, wheel orientation, and engine and transmission values. Some sensors detect the position or orientation of the vehicle, such as global positioning system (GPS) sensors; accelerometers, such as piezoelectric or micro-electromechanical systems (MEMS); gyroscopes, such as rate gyroscopes, ring laser gyroscopes, or fiber optic gyroscopes; inertial measurement units (IMUs); and magnetometers. Some sensors detect the external world, such as radar sensors, scanning laser rangefinders, light detection and ranging (lidar) devices, and image processing sensors such as cameras. Summary of the invention

[0003] The systems and techniques described herein can provide anonymization of personally identifiable information (PII) in image frames while retaining the usefulness of the image frames for performing or training machine learning perception algorithms (such as object detection). Examples of personally identifiable information (PII) in image data include facial images, images of logos or text (such as license plates), etc. Removing PII from image data may be more respectful of personal privacy, but the removal may also remove data useful for training or performing perception algorithms. As described in the present disclosure, a computer is programmed to identify PII in an initial image frame; encode a region of the initial image frame containing the PII into a latent vector having a lower dimension than the region; anonymize the region from the initial image frame, thereby generating a modified image frame; and associate the latent vector with the modified image frame. The latent vector can save such data, for example, identifying an object as a face or a license plate, the orientation of the face, the color scheme of the license plate, etc., thereby saving the data with the image frame that can be used for perception tasks. At the same time, the technology herein can remove features that can be used, for example, to uniquely identify a specific individual. Thus, our techniques provide anonymization of PII from image data as well as robust performance of perceptual algorithms using image data.

[0004] A computer includes a processor and a memory, and the memory stores instructions executable by the processor to identify personally identifiable information (PII) in an initial image frame; encode a region of the initial image frame containing the PII into a latent vector having a lower dimensionality than the region; anonymize the region from the initial image frame to produce a modified image frame; and associate the latent vector with the modified image frame.

[0005] In one example, the instructions may further include instructions for actuating a component of the vehicle based on the modified image frame having the latent vector. In yet another example, the instructions may further include instructions for receiving an initial image frame from a camera of the vehicle.

[0006] In yet another example, the instructions may further include instructions for executing a machine learning perception algorithm using the modified image frame having the latent vector as input, and actuating a component based on an output of the machine learning perception algorithm.

[0007] In one example, the instructions may further include instructions for executing a machine learning perception algorithm using the modified image frame with the latent vector as input. In another example, the instructions may further include instructions for executing a machine learning perception algorithm using the initial image frame and the modified image frame with the latent vector as input.

[0008] In one example, the instructions may also include instructions for embedding the latent vector into the modified image frame.

[0009] In one example, the modified image frame with the latent vector may have the same dimensionality as the initial image frame.

[0010] In one example, the instructions may further include instructions for transmitting the modified image frame having the latent vector to a remote server. In yet another example, the instructions may further include instructions for avoiding transmitting the initial image frame to the remote server.

[0011] In one example, PII may include a person's face. In yet another example, a latent vector may encode non-personally identifiable data about a face.

[0012] In one example, PII may include a depiction of text.

[0013] In one example, the instructions may also include instructions for performing the following operations: executing a machine learning encoder on the PII to encode the PII into a latent vector. In another example, the machine learning encoder may be trained as part of an encoder-decoder architecture to reconstruct a region of an initial image frame. In yet another example, the encoder-decoder architecture may be trained using a loss function that includes a reconstruction loss that measures the similarity between an output of the encoder-decoder architecture and a region of an initial image frame input to the encoder-decoder architecture.

[0014] In yet another example, the instructions may further include instructions for executing a machine learning perception algorithm using the modified image frame having the latent vector as input, and the machine learning perception algorithm may be jointly trained with the machine learning encoder. In yet another example, the machine learning perception algorithm may be jointly trained with the machine learning encoder, wherein the latent vector output by the machine learning encoder is input to the machine learning perception algorithm.

[0015] In yet another example, a machine learning perception algorithm and a machine learning encoder may be jointly trained using a loss function that includes a loss that measures the accuracy of correctly identifying objects in an input image frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram of an exemplary vehicle in communication with a remote server.

[0017] Figure 2 are depictions of exemplary image frames before and after removing personally identifiable information and encoding the latent vector.

[0018] Figure 3 is an illustration of an exemplary machine learning architecture on a vehicle.

[0019] Figure 4 is an illustration of an exemplary training architecture for training a machine learning architecture.

[0020] Figure 5 is a flow chart of an exemplary process for controlling a vehicle with a machine learning architecture. DETAILED DESCRIPTION

[0021] Referring to the accompanying drawings, in which like numbers indicate like components in several views, a computer 105, 145 includes a processor and a memory, and the memory stores instructions executable by the processor to identify personally identifiable information (PII) 205 in an initial image 200; encode a region 210 of the initial image frame 200 containing the PII 205 into a latent vector 220 having a lower dimension than the region 210; anonymize the region 210 from the initial image frame 200 to produce a modified image frame 225; and associate the latent vector 220 with the modified image frame 225.

[0022] refer to Figure 1, the techniques herein may be partially or completely executed on a vehicle 100. The vehicle 100 may be any passenger vehicle or commercial vehicle, such as a car, truck, sport utility vehicle, crossover, van, minivan, taxi, bus, etc. The vehicle 100 may include a vehicle computer 105, a communication network 110, a camera 115, a propulsion system 120, a braking system 125, a steering system 130, a user interface 135, and a transceiver 140.

[0023] The vehicle computer 105 is a microprocessor-based computing device, such as a general-purpose computing device (including a processor and memory, an electronic controller, etc.), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a combination of the foregoing, etc. In general, hardware description languages ​​such as VHDL (VHSIC (Very High Speed ​​Integrated Circuit) Hardware Description Language) are used in electronic design automation to describe digital and mixed signal systems such as FPGAs and ASICs. For example, ASICs are manufactured based on VHDL programming provided before manufacturing, and the logic components inside the FPGA can be configured based on, for example, VHDL programming stored in a memory electrically connected to the FPGA circuit. Therefore, the vehicle computer 105 may include a processor, a memory, etc. The memory of the vehicle computer 105 may include a medium for storing instructions executable by the processor and for electronically storing data and / or a database, and / or the vehicle computer 105 may include a structure such as the aforementioned structure for providing programming. The vehicle computer 105 may be a plurality of computers coupled together on the vehicle 100.

[0024] The vehicle computer 105 can transmit and receive data through the communication network 110. The communication network 110 can be, for example, a controller area network (CAN) bus, Ethernet, WiFi, a local interconnect network (LIN), an on-board diagnostic connector (OBD-II), and / or any other wired or wireless communication network. The vehicle computer 105 can be communicatively coupled to the camera 115, the propulsion system 120, the braking system 125, the steering system 130, the user interface 135, the transceiver 140, and other components of the vehicle 100 via the communication network 110.

[0025] The camera 115 may detect electromagnetic radiation within a range of wavelengths. For example, the camera 115 may detect visible light, infrared radiation, ultraviolet light, or a range of wavelengths including visible light, infrared light, and / or ultraviolet light. For example, the camera 115 may be a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or any other suitable type. For another example, the camera 115 may be a time of flight (TOF) camera that includes a modulated light source for illuminating an environment and detects both reflected light from the modulated light source and ambient light to sense reflectivity amplitude and distance to a scene.

[0026] The propulsion system 120 of the vehicle 100 generates energy and converts the energy into motion of the vehicle 100. The propulsion system 120 may be a conventional vehicle propulsion subsystem, such as a conventional powertrain system including an internal combustion engine coupled to a transmission that transfers rotational motion to the wheels; an electric powertrain system including a battery, an electric motor, and a transmission that transfers rotational motion to the wheels; a hybrid powertrain system including elements of a conventional powertrain system and an electric powertrain system; or any other type of propulsion device. The propulsion system 120 may include an electronic control unit (ECU) that communicates with and receives input from the vehicle computer 105 and / or a human operator, etc. The human operator may control the propulsion system 120 via, for example, an accelerator pedal and / or a gear shift lever.

[0027] The brake system 125 is typically a conventional vehicle braking subsystem and resists the motion of the vehicle 100, thereby slowing and / or stopping the vehicle 100. The brake system 125 may include friction brakes, such as disc brakes, drum brakes, band brakes, etc.; regenerative brakes; any other suitable type of brake; or a combination thereof. The brake system 125 may include an electronic control unit (ECU) or the like that communicates with and receives input from the vehicle computer 105 and / or a human operator. The human operator may control the brake system 125 via, for example, a brake pedal.

[0028] The steering system 130 is typically a conventional vehicle steering subsystem and controls the turning of the wheels. The steering system 130 may be a rack and pinion system with electric power steering, a steer-by-wire system (both of which are known), or any other suitable system. The steering system 130 may include an electronic control unit (ECU) or the like that communicates with and receives input from the vehicle computer 105 and / or a human operator. The human operator may control the steering system 130 via, for example, a steering wheel.

[0029] The user interface 135 presents information to and receives information from an operator of the vehicle 100. The user interface 135 may be located, for example, on a dashboard in the passenger compartment of the vehicle 100, or anywhere else that the operator can easily see it. The user interface 135 may include a dial, a digital readout, a screen, a speaker, etc., for providing information to the operator, such as, for example, known human-machine interface (HMI) elements. The user interface 135 may include buttons, knobs, keypads, microphones, etc., for receiving information from the operator.

[0030] The transceiver 140 may be adapted to communicate via any suitable wireless communication protocol (such as cellular, The transceiver 140 may be adapted to communicate with a remote server 145 (i.e., a server that is different and spaced apart from the vehicle 100). The transceiver 140 may be one device, or may include separate transmitters and receivers.

[0031] The remote server 145 may be located external to the vehicle 100. For example, the remote server 145 may be associated with another vehicle (e.g., V2V communication), associated with an infrastructure component (e.g., V2I communication), associated with an emergency responder, associated with a mobile device associated with an owner of the vehicle 100, a fleet manager or manufacturer of the vehicle 100, and the like. The remote server 145 is a microprocessor-based computing device, such as a general-purpose computing device including a processor and a memory. The memory of the remote server 145 may include a medium for storing instructions executable by the processor and for electronically storing data and / or databases, and / or the remote server 145 may include a structure such as the aforementioned structure that provides programming. The remote server 145 may be a plurality of computers coupled together.

[0032] Unless otherwise noted, the techniques described below can be performed by the vehicle computer 105, the remote server 145, or a combination of the vehicle computer 105 and the remote server 145, for example, an earlier step performed by the vehicle computer 105 and a later step performed by the remote server 145. In any step taken over by the remote server 145, the vehicle computer 105 can instruct the transceiver 140 to transmit the necessary data from the earlier step to the remote server 145. If a step can be performed by the vehicle computer 105 or the remote server 145, the present disclosure refers to them collectively as computers 105, 145. The advantage of performing the steps on the vehicle computer 105 is that the PII 205 is not transmitted from the vehicle 100.

[0033] refer to Figure 2, the computer 105, 145 is programmed to receive an initial image frame 200 from the camera 115, for example, image data including a plurality of initial image frames 200. The term "initial" indicates an image frame before or without modification as described herein. The image data is a series of initial image frames 200 of the field of view of the camera 115. Each initial image frame 200 is a two-dimensional matrix of pixels. The brightness or color of each pixel is represented as one or more numerical values, for example, a scalar unitless value of photometric light intensity between 0 (black) and 1 (white), or a value of each of red, green, and blue, for example, each in an 8-bit scale (0 to 255) or a 12-bit or 16-bit scale. The pixels may be a mixture of representations, such as a repeating pattern of scalar values ​​of the intensity of three pixels and a fourth pixel having three numerical color values, or some other pattern. The location in the image frame (i.e., the location in the field of view of the sensor when the image frame was recorded) can be specified in pixel dimensions or coordinates, for example, a pair of ordered pixel distances, such as a number of pixels from the top edge of the image frame and a number of pixels from the left edge of the image frame. As an alternative to receiving the initial image frame 200 from the camera 115, the initial image frame 200 can be received from a third party as part of a previously generated image data set.

[0034] Initial image frame 200 includes personally identifiable information (PII) 205, for example, someone might be able to uniquely identify an individual from data in the initial image frame, for example, using an algorithm such as facial recognition or optical character recognition. For purposes of this disclosure, "personally identifiable information" is defined as information representation that allows a reasonable inference to be made about the identity of the individual to whom the information applies. For example, while vehicle 100 is traveling, initial image frame 200 may include the faces of people (e.g., pedestrians) near vehicle 100. For another example, initial image frame 200 may include a depiction of text on a license plate of another vehicle, such as Figure 2 as shown in .

[0035] The computer 105, 145 may be programmed to identify the PII 205 in the initial image frame 200. For example, the computer 105, 145 may identify the PII 205 using conventional image recognition techniques (e.g., a convolutional neural network) that is programmed to accept the initial image frame 200 as input and output at least one classification of the objects in the initial image frame 200. For example, the output may be a list of bounding boxes surrounding objects with a particular classification (e.g., “face”) and a corresponding confidence level for that classification applied to the respective objects, e.g., {x 左1 ,y 顶部1 ,x 右1 ,y 底部1，95 %;x 左2 ,y 顶部2 , x 右2 ,y底部2，87 %; ...}. The bounding box is described in more detail below. The convolutional neural network includes a series of layers, each of which uses the previous layer as input. Each layer contains a plurality of neurons that receive data generated by a subset of the neurons of the previous layer as input and generate outputs that are sent to the neurons in the next layer. Types of layers include: convolutional layers, which calculate the dot product of weights and input data of a small area; pooling layers, which perform downsampling operations along the spatial dimension; and fully connected layers, which are generated based on the outputs of all neurons in the previous layer. The last layer of the convolutional neural network generates a score for each potential classification of the object, and the final output is the classification with the highest score. Alternatively, if the potential classification that is the type of PII 205 has a score above a threshold, the final output can be the classification even if the non-PII potential classification has a higher score. If the classification is the type of PII 205, the computer 105, 145 has identified PII 205.

[0036] For another example, the computer 105, 145 may use an algorithm that is specialized for a particular type of PII 205 (e.g., a face or a license plate) to identify the PII 205 in the initial image frame 200. For a face, as an example, the computer 105, 145 may detect a face in the initial image frame 200, for example, by using any suitable face detection technique, such as: a knowledge-based technique, such as a multi-resolution rule-based approach; a feature-invariant technique, such as edge grouping, a spatial gray-level correlation matrix, or a Gaussian mixture; a template matching technique, such as a shape template or an active shape model; a cascade classifier, such as a Haar feature; or an appearance-based technique, such as eigenface decomposition and clustering, a Gaussian distribution and a multilayer perceptron, a neural network, a support vector machine with a polynomial kernel, a naive Bayes classifier with joint statistics of local appearance and location, a high-order statistics with a hidden Markov model, or Kullback correlation information.

[0037] Additionally, the computer 105, 145 may identify PII 205 in the initial image frame 200 based on identification of PII 205 in earlier or later image frames. For example, the computer 105, 145 may execute a Kalman filter or a particle filter to identify PII 205 in image frames where the above-described classification techniques have temporarily fallen below a confidence threshold for identification.

[0038] The computer 105, 145 may be programmed to define an area 210 that contains the PII 205. For example, the area 210 may be defined by a bounding box 215 that surrounds the corresponding area 210. The computer 105, 145 may be programmed to generate a corresponding bounding box 215 around the identified PII 205. Each bounding box 215 may be defined by, for example, pixel coordinates of opposing corners of the bounding box 215, thereby specifying a rectangle. For example, the computer 105, 145 may generate a bounding box 215 around the content that the computer 105, 145 identifies as the PII 205. The computer 105, 145 may generate each bounding box 215 as a minimum size that contains the corresponding PII 205, for example, by constructing a pixel coordinate pair for the bounding box 215 using the highest and lowest vertical pixel coordinates and the leftmost and rightmost horizontal pixel coordinates of the area 210.

[0039] The computer 105, 145 is programmed to encode a region 210 containing PII 205 into a latent vector 220. The latent vector 220 holds certain information about the object with PII 205. The term "latent vector" is used in its machine learning sense to represent a subset of input data in a more compact form than the input data. In the present application, the latent vector 220 has a lower dimensionality than the region 210 encoded into the latent vector 220. For example, a 100 pixel x 100 pixel region 210 may be encoded into a 15 x 1 latent vector 220. The latent vector 220 may encode non-personally identifiable data about an object with PII 205 (e.g., about a face or license plate). For example, a latent vector 220 for a face may encode the fact that the object includes a face, the orientation of the face (i.e., which direction the face is facing), whether the eyes are open or closed, which direction the eyes are looking, facial expression, etc., and the latent vector 220 for a face may lack dimensions associated with features of the face from which a facial recognition algorithm or a human observer (once the image is reconstructed) may be able to uniquely identify the face. For another example, a latent vector 220 for a license plate may encode the fact that the object is a license plate, the color scheme (e.g., the color of the background and letters), etc., and the latent vector 220 for a license plate may lack the specific characters displayed on the license plate. (This will be discussed below with respect to Figure 3 An algorithm for encoding region 210 into latent vector 220 is discussed. )

[0040] The computer 105, 145 is programmed to anonymize the region 210 from the initial image frame 200, thereby generating a modified image frame 225. For example, the computer 105, 145 may anonymize the region 210 by using data not derived from the region 210 (e.g., Figure 2The region 210 may be deleted from the initial image frame 200 by overwriting the pixels of the region 210 with a pixel (shown as solid black) in the original image frame 200. For another example, the computer 105, 145 may blur the region 210 of the initial image frame 200. Blurring the region 210 may be based on the content of the region 210. For example, the computer 105, 145 may use any suitable blurring technique that transforms the content of the region 210, such as a Gaussian blur. Advantageously, the modified image frame 225 thus lacks the PII 205.

[0041] The computer 105, 145 is programmed to associate the latent vector 220 with the modified image frame 225. For example, the computer 105, 145 may store the latent vector 220 as part of the modified image frame 225, e.g., embed the latent vector 220 in the modified image frame 225 (e.g., Figure 2 205), or stored as metadata of the modified image frame 225. The modified image frame 225 with the latent vector 220 has the same dimensionality as the initial image frame 200. Specifically, the latent vector 220 can be embedded into a sub-portion of the region 210 of the modified image frame 225 that previously contained the PII 205, for example, as part of the overwrite just described above, where the padding 230 is used for the rest of the pixels of the region 210. As part of the embedding, the computer 105, 145 can project the latent vector down to the image dtype (i.e., data type object) of the modified image frame 225, for example, int8, int12, etc. Since the dimensionality of the latent vector 220 is lower than that of the region 210, the space occupied by the latent vector 220 will be less than the entire region 210. For another example, the computer 105, 145 can store the latent vector 220 in a file or database that references the index of the modified image frame 225 and the pixel coordinates of the region 210 within the modified image frame 225. The index may specify the particular camera 115 that generated the corresponding initial image frame 200 and the time at which the corresponding initial image frame 200 was generated. By associating the latent vector 220 with the modified image frame 225, the computer 105, 145 may provide the modified image frame 225 and the latent vector 220 together as input to an algorithm for which the latent vector 220 provides useful information, or the vehicle computer 105 may transmit the modified image frame 225 and the latent vector 220 together to a remote server 145, as described below. The computer 105, 145 may also associate a bounding box 215 defining the area 210 with the modified image frame 225, thereby providing identification of the location in the modified image frame 225 where anonymization occurred.

[0042] The vehicle computer 105 may be programmed to transmit the modified image frame 225 with the latent vector 220 (and possibly the bounding box 215) to the remote server 145. For example, the vehicle computer 105 may instruct the transceiver 140 to transmit a message containing the modified image frame 225 and the latent vector 220 to the remote server 145. The vehicle computer 105 may be programmed to avoid transmitting the initial image frame 200 to the remote server 145, thereby not transmitting the PII 205 outside the vehicle 100.

[0043] From remote server 145, modified image frame 225 and latent vector 220 can be used as training data or test data for machine learning algorithms (e.g., research, validation, verification, etc. of ADAS features). (Such use is distinct from the machine learning algorithms described below.) Beneficially, modified image frame 225 lacks PII 205, while latent vector 220 still contains data that the algorithm would have derived from PII 205. Such data can be used, for example, by using techniques such as generative neural networks to regenerate a generic version of PII 205, e.g., a face that is different from the face that appeared in the original image frame 200. Thus, the combination of modified image frame 225 and latent vector 220 serves as more accurate training data or test data than image data that has had PII 205 anonymized without the addition of latent vector 220.

[0044] refer to Figure 3, the computer 105, 145 can store the machine learning architecture 300. The computer 105, 145 can use the initial image frame 200 and the modified image frame 225 with the latent vector 220 as input to execute the machine learning perception algorithm 305 of the machine learning architecture 300. Executing the machine learning perception algorithm 305 using both the initial image frame 200 and the modified image frame 225 with the latent vector 220 can improve the accuracy of the machine learning perception algorithm 305. For example, the machine learning architecture 300 can include a first instance 305a of the machine learning perception algorithm 305, a machine learning encoder 310, a second instance 305b of the machine learning perception algorithm 305, and a fusion algorithm 315. The first instance 305a of the machine learning perception algorithm 305 can take the initial image frame 200 as input and output a first recognition of an object in the field of view of the camera 115. The machine learning encoder 310 can encode the region 210 of the initial image frame 200 containing the PII 205 into the latent vector 220. In conjunction with the above-described steps of anonymizing the region 210 to arrive at the modified image frame 225 and associating the latent vector 220 with the modified image frame 225, the machine learning encoder 310 produces the modified image frame 225 with the latent vector 220. The first instance 305b of the machine learning perception algorithm 305 can take the modified image frame 225 with the latent vector 220 as input and output a second recognition of the object in the field of view of the camera 115. The fusion algorithm 315 can combine the first recognition and the second recognition into a final recognition. The final recognition from the machine learning architecture 300 can be used to actuate the vehicle 100 in the actuation step 320. By using an alternative analysis of the initial image frame 200 (i.e., the modified image frame 225 with the latent vector 220), the machine learning architecture 300 can provide a more accurate recognition than using only the initial image frame 200.

[0045] The computer 105, 145 can be programmed to execute a first instance 305a of a machine learning perception algorithm 305 using the initial image frame 200 as input. The machine learning perception algorithm 305 can be, for example, a convolutional neural network programmed to accept an image as input and output a recognition of an object in the image. A convolutional neural network includes a series of layers, each of which uses the previous layer as input. Each layer contains a plurality of neurons that receive data generated by a subset of neurons in the previous layer as input and generate outputs that are sent to neurons in the next layer. Types of layers include: a convolutional layer that calculates a dot product of weights and input data for a small region; a pooling layer that performs a downsampling operation along a spatial dimension; and a fully connected layer that is generated based on the output of all neurons in the previous layer. The last layer of the convolutional neural network generates a score for each potential recognition of an object, and the final output of the object is the recognition with the highest score. The recognition from the first instance 305a will be referred to as the "first recognition".

[0046] The computer 105, 145 may be programmed to encode the region 210 of the initial image frame 200 containing the PII 205 into a latent vector 220. For example, the computer 105, 145 may execute a machine learning encoder 310 on the PII 205 to encode the PII 205 into the latent vector 220. The machine learning encoder 310 may be or include an artificial neural network, such as a residual neural network (ResNet). The machine learning encoder 310 may be derived from an encoder-decoder architecture 405, as described below with respect to Figure 4 As described. The machine learning encoder 310 may include a series of layers and may reduce the dimensionality of the data from one layer to the next. The output of the last layer may be the latent vector 220.

[0047] The computer 105, 145 can be programmed to perform a second instance 305b of the machine learning perception algorithm 305 using the modified image frame 225 with the latent vector 220 as input in the same manner as described above for the first instance 305a. The recognition from the second instance 305b will be referred to as the "second recognition." The second recognition of a given object can be the same as or different from the first recognition of the object.

[0048] Computers 105, 145 can be programmed to execute fusion algorithm 315, for example, taking the first identification and the second identification as input. Fusion algorithm 315 can perform sensor fusion. Sensor fusion combines data from different sources so that the uncertainty of the resulting data is lower than using the data from each source alone, for example in terms of creating a unified model of the environment around vehicle 100. Sensor fusion can be performed with one or more algorithms, such as Kalman filter, central limit theorem, Bayesian network, evidence theory, convolutional neural network, etc. As an example, fusion algorithm 315 can use ensemble learning. Ensemble learning combines predictions from multiple algorithms. Computers 105, 145 can use any suitable type of ensemble learning, such as Bayesian optimal classifier, boosting, Bayesian model averaging, Bayesian model combination, model bucket, voting, etc. Fusion algorithm 315 outputs the final identification of the object based on the corresponding first identification and second identification.

[0049] The vehicle computer 105 may be programmed to actuate components of the vehicle 100 based on the modified image frame 225 with the latent vector 220. For example, the vehicle computer 105 may actuate the component based on the final recognition from the fusion algorithm 315. Therefore, the actuation of the component is indirectly based on the output of the machine learning perception algorithm 305 and the modified image frame 225 with the latent vector 220. In other words, the modified image frame 225 with the latent vector 220 may be an input to the software stack that causes the actuation of the component of the vehicle 100. The component may include, for example, the propulsion system 120, the braking system 125, the steering system 130, and / or the user interface 135. For example, the vehicle computer 105 may actuate the component as part of performing an advanced driver assistance system (ADAS). ADAS is an electronic technology that assists the driver in achieving driving functions and parking functions. Examples of ADAS include forward approach detection, lane departure detection, blind spot detection, brake actuation, adaptive cruise control, and lane keeping assist systems. The vehicle computer 105 may actuate the braking system 125 to brake the vehicle 100 or actuate the user interface 135 to output a message to the operator in response to identifying the object as a person at a certain location relative to the vehicle 100. For another example, the vehicle computer 105 may autonomously operate the vehicle 100 based on the environment as detected by the camera 115 and other sensors and the resulting identification, i.e., actuate the propulsion system 120, the braking system 125, and the steering system 130, for example, to navigate around the environment.

[0050] The machine learning architecture 300 may alternatively have Figure 3The different forms shown. For example, the machine learning perception algorithm 305 may include a machine learning encoder 310, for example, within a layer of a convolutional neural network. The computer 105, 145 may execute the machine learning perception algorithm 305 as a single instance using the initial image frame 200 and the modified image frame 225 with the latent vector 220 as input. The recognition provided by the single instance of the machine learning perception algorithm 305 may then be used in the actuation step 320. For another example, the machine learning perception algorithm 305 may perform a task different from recognition, such as depth estimation, object orientation, pedestrian trajectory estimation, etc.

[0051] refer to Figure 4 , the training architecture 400 can be used to train the machine learning perception algorithm 305 and / or the machine learning encoder 310. The machine learning perception algorithm 305 and the machine learning encoder 310 can be jointly trained, wherein the latent vector 220 output by the machine learning encoder 310 is input to the machine learning perception algorithm 305. The machine learning encoder 310 is trained as part of an encoder-decoder architecture 405 including a decoder 410. Training data of the input image frame is provided as input to the machine learning perception algorithm 305 and the encoder-decoder architecture 405, and the corresponding latent vector 220 generated by the machine learning encoder 310 is provided as an additional input to the machine learning perception algorithm 305. The machine learning perception algorithm 305 outputs an identification 420 of the object in the initial image frame 200. The training architecture 400 is trained to correctly identify the object in the input image frame and reconstruct the region 210 of the input image frame containing the PII 205.

[0052] The machine learning encoder 310 is trained as part of the encoder-decoder architecture 405 to reconstruct the region 210 of the initial image frame 200. The encoder-decoder architecture 405 includes the machine learning encoder 310, followed by the decoder 410. The decoder 410 takes the output of the machine learning encoder 310 (i.e., the latent vector 220) as input. The decoder 410 can be or include an artificial neural network, such as ResNet. The decoder 410 can include a series of layers and can increase the dimensionality of the data from one layer to the next. The output of the last layer can be a reconstruction 415 of the region 210 containing the PII 205. The reconstruction 415 can have the same dimensionality as the region 210.

[0053] The machine learning perception algorithm 305 is jointly trained with the encoder-decoder architecture 405. The machine learning perception algorithm 305 and the encoder-decoder architecture 405 are interconnected as described above, receive the same training data, and are updated using the same loss function (described below). For example, as is known, back propagation can be used to update the weights of the connections between the layers of the machine learning perception algorithm 305 and the encoder-decoder architecture 405 to minimize the value of the loss function. Joint training helps the machine learning encoder 310 generate a latent vector 220 with information useful to the machine learning perception algorithm 305, and joint training helps the machine learning perception algorithm 305 use the latent vector 220 to improve the accuracy of the recognition 420.

[0054] The loss function may include a reconstruction loss and / or a recognition loss. The reconstruction loss measures the similarity between the output of the encoder-decoder architecture 405 (i.e., the reconstruction 415) and the region 210 of the initial image frame 200 input to the encoder-decoder architecture 405. The similarity may be expressed at the pixel level, at the feature level, or in some other aspect. For the purposes of this disclosure, the term "feature" is used in its computer vision sense as a piece of information about the content of the image, specifically whether a certain area of ​​the image has certain attributes. The recognition loss measures the accuracy of correctly producing the recognition 420 of the object in the input image frame. The training data may include annotations giving ground truth recognitions, and the recognition 420 from the machine learning perception algorithm 305 may be compared with the annotations. For example, the loss function may be the sum of the reconstruction loss and the recognition loss.

[0055] Figure 5 is a flow chart illustrating an exemplary process 500 for controlling a vehicle 100 having a machine learning architecture 300. The memory of the computer 105, 145 stores executable instructions for performing the steps of the process 500, and / or may be programmed such as the structures mentioned above. Specifically, the process 500 may be performed entirely by the vehicle computer 105. As a general overview of the process 500, the vehicle computer 105 receives an initial image frame 200, identifies PII 205 in the initial image frame 200, encodes the PII 205 into a latent vector 220, and anonymizes the region 210 from the initial image frame 200 to associate the latent vector 220 with the now modified image frame 225, transmits the modified image frame 225 with the latent vector 220 to the remote server 145, executes the machine learning perception algorithm 305, and actuates components of the vehicle 100.

[0056] Process 500 begins at block 505 where the vehicle computer 105 receives an initial image from the camera 115 , as described above.

[0057] Next, in block 510 , the vehicle computer 105 identifies the PII 205 in the initial image frame 200 , as described above.

[0058] Next, in block 515 , the vehicle computer 105 encodes the PII 205 into a latent vector 220 , anonymizes the region 210 containing the PII 205 from the initial image frame 200 to obtain a modified image frame 225 , and associates the latent vector 220 with the modified image frame 225 , as described above.

[0059] Next, in block 520 , the vehicle computer 105 transmits the modified image frame 225 with the latent vector 220 to the remote server 145 , as described above.

[0060] Next, in box 530, the vehicle computer 105 uses the modified image frame 225 with the latent vector 220 as input to execute the machine learning perception algorithm 305, for example, as part of executing the machine learning architecture 300. The vehicle computer 105 thereby generates an identification of the object in the initial image frame 200.

[0061] Next, in block 535, the vehicle computer 105 actuates a component of the vehicle 100 based on the output of the machine learning architecture 300 (e.g., the identification of the object in the initial image frame 200), as described above. Thus, the actuation of the component is indirectly based on the modified image frame 225 having the latent vector 220. After block 535, the process 500 ends.

[0062] In general, the computing systems and / or devices described may employ any of a number of computer operating systems, including but not limited to versions and / or variations of: Ford Applications, AppLink / Smart Device Link middleware, Microsoft Operating system, Microsoft Operating systems, Unix operating systems (e.g., those distributed by Oracle Corporation of Redwood Shores, California) operating systems), AIX UNIX operating system released by International Business Machines Corporation of Armonk, New York, Linux operating system, Mac OSX and iOS operating systems released by Apple Inc. of Cupertino, California, BlackBerry OS released by BlackBerry Ltd. of Waterloo, Canada, and Android operating system developed by Google Inc. and the Open Handset Alliance, or provided by QNX Software Systems CAR Infotainment Platform. Examples of computing devices include, but are not limited to, an in-vehicle computer, a computer workstation, a server, a desktop, notebook, laptop or handheld computer, or some other computing system and / or device.

[0063] Computing devices typically include computer-executable instructions, wherein the instructions are executable by one or more computing devices such as those listed above. Computer-executable instructions may be compiled or interpreted from a computer program created using a variety of programming languages ​​and / or technologies, including, alone or in combination, but not limited to Java. TM , C, C++, Matlab, Simulink, Stateflow, Visual Basic, Java Script, Python, Perl, HTML, etc. Some of these applications can be compiled and executed on virtual machines such as Java virtual machines, Dalvik virtual machines, etc. Typically, a processor (e.g., a microprocessor) receives instructions from, for example, a memory, a computer-readable medium, etc., and executes these instructions to perform one or more processes, including one or more of the processes described herein. Such instructions and other data can be stored and transmitted using a variety of computer-readable media. Files in a computing device are typically a collection of data stored on a computer-readable medium such as a storage medium, a random access memory, etc.

[0064] Computer-readable media (also referred to as processor-readable media) include any non-transitory (e.g., tangible) media that participate in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take many forms, including, but not limited to, non-volatile media and volatile media. Instructions can be transmitted via one or more transmission media, including optical fibers, wires, wireless communications, including internals that make up a system bus coupled to a processor of a computer. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, FLASH-EEPROM, any other memory chip or cassette, or any other medium from which a computer can read.

[0065] The database, data repository or other data storage described herein may include various mechanisms for storing, accessing / accessing and retrieving various data, including hierarchical databases, file sets in file systems, application databases in special formats, relational database management systems (RDBMS), non-relational databases (NoSQL), graphic databases (GDB), vector databases, etc. Each such data storage is typically included in a computing device using a computer operating system such as one of the above-mentioned, and is accessed via a network in any one or more of a variety of ways. The file system can be accessed from the computer operating system and may include files stored in various formats. In addition to the language (such as the above-mentioned PL / SQL language) for creating, storing, editing and executing stored programs, RDBMS typically also uses structured query language (SQL).

[0066] In some examples, system elements may be implemented as computer-readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.) stored on computer-readable media associated therewith (e.g., disks, memories, etc.). A computer program product may include such instructions stored on a computer-readable medium for performing the functions described herein.

[0067] In the accompanying drawings, like reference numerals indicate like elements. In addition, some or all of these elements may be changed. With respect to the media, processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of such processes, etc. have been described as occurring in a certain ordered sequence, such processes may be practiced by performing the steps in an order different from that described herein. It should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. The operations, systems, and methods described herein should always be implemented and / or performed in accordance with applicable owner / user manuals and / or safety guidelines.

[0068] The present disclosure has been described in an illustrative manner, and it should be understood that the terms that have been used are intended to be in the nature of descriptive words, rather than restrictive. The adjectives "first" and "second" are used throughout this document as identifiers, and are not intended to indicate importance, order, or quantity. The use of "in response to," "after determining...", etc. indicates a causal relationship, not just a temporal relationship. In view of the above teachings, many modifications and variations of the present disclosure are possible, and the present disclosure can be practiced in other ways than those specifically described.

[0069] According to the present invention, a computer is provided, which has a processor and a memory, and the memory stores instructions, which can be executed by the processor to: identify personally identifiable information (PII) in an initial image frame; encode a region of the initial image frame containing the PII into a latent vector having a lower dimension than the region; anonymize the region from the initial image frame to produce a modified image frame; and associate the latent vector with the modified image frame.

[0070] According to one embodiment, the instructions further include instructions for actuating a component of a vehicle based on the modified image frame having the latent vector.

[0071] According to one embodiment, the instructions further comprise instructions for receiving the initial image frame from a camera of the vehicle.

[0072] According to one embodiment, the instructions further include instructions for executing a machine learning perception algorithm using the modified image frame with the latent vector as input, and actuating the component based on an output of the machine learning perception algorithm.

[0073] According to one embodiment, the instructions further include instructions for executing a machine learning perception algorithm using the modified image frame with the latent vector as input.

[0074] According to one embodiment, the instructions further include instructions for executing a machine learning perception algorithm using the initial image frame and the modified image frame with the latent vector as input.

[0075] According to one embodiment, the instructions further include instructions for embedding the latent vector in the modified image frame.

[0076] According to one embodiment, the modified image frame with the latent vector has the same dimensionality as the initial image frame.

[0077] According to one embodiment, the instructions further include instructions for transmitting the modified image frame with the latent vector to a remote server.

[0078] According to one embodiment, the instructions may further include instructions for avoiding transmitting the initial image frame to the remote server.

[0079] According to one embodiment, the PII includes a person's face.

[0080] According to one embodiment, the latent vector encodes non-personally identifiable data about the face.

[0081] According to one embodiment, the PII includes a depiction of text.

[0082] According to one embodiment, the instructions further include instructions for executing a machine learning encoder on the PII to encode the PII into the latent vector.

[0083] According to one embodiment, the machine learning encoder is trained as part of an encoder-decoder architecture to reconstruct the region of the initial image frame.

[0084] According to one embodiment, the encoder-decoder architecture is trained using a loss function comprising a reconstruction loss measuring the similarity between the output of the encoder-decoder architecture and the region of the initial image frame input to the encoder-decoder architecture.

[0085] According to one embodiment, the instructions also include instructions for executing a machine learning perception algorithm using the modified image frame with the latent vector as input, and jointly training the machine learning perception algorithm with the machine learning encoder.

[0086] According to one embodiment, the machine learning perception algorithm and the machine learning encoder are jointly trained, wherein the latent vector output by the machine learning encoder is input to the machine learning perception algorithm.

[0087] According to one embodiment, the training of the machine learning perception algorithm and the machine learning encoder are jointly performed using a loss function, the loss function comprising a loss that measures the accuracy of correctly identifying objects in an input image frame.

[0088] According to the present invention, a method includes: identifying personally identifiable information (PII) in an initial image frame; encoding a region of the initial image frame containing the PII as a latent vector having a lower dimension than the region; anonymizing the region from the initial image frame to produce a modified image frame; and associating the latent vector with the modified image frame.

Claims

1. A method comprising: identifying personally identifiable information (PII) in the initial image frame; encoding a region of the initial image frame containing the PII into a latent vector having a lower dimension than the region; anonymizing the region from the initial image frame, thereby producing a modified image frame; as well as The latent vector is associated with the modified image frame. 2 . The method of claim 1 , further comprising actuating a component of a vehicle based on the modified image frame having the latent vector. 3 . The method of claim 2 , further comprising receiving the initial image frame from a camera of the vehicle.

4. The method of claim 2, further comprising: performing a machine learning perception algorithm using the modified image frame with the latent vector as input; as well as The component is actuated based on the output of the machine learning perception algorithm.

5. The method of claim 1 further comprising performing a machine learning perception algorithm using the modified image frame with the latent vector as input.

6. The method of claim 1 further comprising embedding the latent vector into the modified image frame.

7. The method of claim 1, wherein the modified image frame having the latent vector has the same dimensionality as the initial image frame.

8. The method of claim 1, further comprising transmitting the modified image frame with the latent vector to a remote server.

9. The method of claim 8, further comprising avoiding transmitting the initial image frame to the remote server.

10. The method of claim 1, wherein the PII comprises a person's face and the latent vector encodes non-personally identifiable data about the face.

11. The method of claim 1, wherein the PII comprises a depiction of text.

12. The method of claim 1, further comprising performing a machine learning encoder on the PII to encode the PII into the latent vector.

13. The method of claim 12, wherein the machine learning encoder is trained as part of an encoder-decoder architecture to reconstruct the region of the initial image frame.

14. The method of claim 12, further comprising executing a machine learning perception algorithm using the modified image frame with the latent vector as input, and jointly training the machine learning perception algorithm with the machine learning encoder.

15. A computer comprising a processor and a memory, the memory storing instructions executable by the processor to perform the method of one of claims 1 to 14.

Images