Lossless privacy preserving data anonymization for ADAS / AD model development

By identifying and replacing advanced attributes of the vehicle and human face and license information, and generating replacement images, the problem of incomplete privacy protection in the prior art is solved, and the implementation of the self-driving vehicle model training while protecting privacy is achieved. The greatest value of the dataset.

CN119067836APending Publication Date: 2024-12-03GM GLOBAL TECHNOLOGY OPERATIONS LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311389446.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-02
Filing Date
2023-10-25
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, when using observation data to assist in the development and training of autonomous driving vehicles, it is difficult to effectively protect individual privacy, especially the protection of facial and license information is not thorough enough, which affects the use value of data.

Method used

By identifying the region of interest, extracting advanced attributes and generating replacement images, protecting individual privacy while maintaining advanced attributes of license plates and faces, only when statistical distribution matching is determined between replacement images and other actual images, ensuring the generated data set Used for privacy protection when training models.

Benefits of technology

It achieves the maximum value of the data set while protecting individual privacy, ensuring that the data is used to assist and autonomous vehicle model training, meeting legal and regulatory requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067836A_ABST
    Figure CN119067836A_ABST
Patent Text Reader

Abstract

Methods and systems for data processing including a camera and a processor are provided. The camera is configured to obtain data including a frame. The processor is coupled to the camera and is configured to at least facilitate: identifying, for each frame, a region of interest for privacy protection; for each region of interest, extracting an advanced attribute of the region of interest of the frame; generating a replacement image for each region of interest, thereby protecting privacy while maintaining advanced attributes of the region of interest; and updating the frame with the replacement image of each of the one or more regions of interest, generating an updated frame that protects individual privacy.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The technical field generally relates to data collection, including personal or private information, and includes model development for assisted and autonomous vehicles. Background Art

[0002] Observational data is often needed or useful for the development and training of assisted and autonomous vehicles, among various other applications. In some cases, such observational data may include personal information of one or more individuals, such as a face or license plate belonging to the individual. However, existing technologies may not always be optimal, for example in terms of utilizing the data in an optimal manner while also protecting the privacy of the individuals.

[0003] Therefore, it is desirable to provide improved methods and systems for generating data that may include personal information, and including model development and training for assisted and autonomous vehicles. In addition, other desirable features and characteristics of the present invention will become apparent from the subsequent detailed description of the present invention and the appended claims, taken in conjunction with the accompanying drawings and this background technology. Summary of the invention

[0004] In an exemplary embodiment, a method is provided, which includes: obtaining data from one or more cameras, the data including one or more frames; identifying one or more regions of interest for privacy protection for each of the one or more frames via one or more processors; extracting high-level properties of the region of interest of the frame for each of the one or more regions of interest via one or more processors; generating a replacement image for each of the one or more regions of interest via one or more processors to protect privacy while maintaining the high-level properties of the region of interest; and updating the frame using the replacement image of each of the one or more regions of interest via one or more processors for each of the one or more frames to generate an updated frame that protects individual privacy.

[0005] Furthermore in accordance with an exemplary embodiment, the one or more regions of interest relate to a license plate of the vehicle; and the replacement image includes a composite license plate image having high level attributes of the license plate.

[0006] Furthermore, according to an exemplary embodiment, the composite license plate image has the same font, color, style, and jurisdiction as the license plate, but includes a different license plate number.

[0007] Furthermore, according to an exemplary embodiment, the method further comprises comparing, via one or more processors, the replacement image with a plurality of actual license plate images from other vehicles; wherein the replacement image is utilized only when a statistically distributed match is determined between the replacement image and the plurality of actual license plate images from other vehicles.

[0008] Further in accordance with an exemplary embodiment, the one or more regions of interest relate to a person's face; and the replacement image includes a synthesized human facial image having high-level attributes of the person's face.

[0009] Furthermore, according to an exemplary embodiment, the synthesized human face image has the same age, race, and size as the face of a person, but the person cannot be identified from the synthesized human face image.

[0010] In addition, according to an exemplary embodiment, the method also includes: comparing the replacement image with a plurality of actual facial images from other persons via one or more processors; wherein the replacement image is utilized only when a match in statistical distribution is determined between the replacement image and the plurality of actual facial images from other persons.

[0011] Furthermore, according to an exemplary embodiment, the data includes video camera images of vehicles and persons along one or more roads on which the vehicle travels.

[0012] Furthermore, according to an exemplary embodiment, the method further comprises: updating one or more data sets in a computer memory using an updated frame of each of the one or more frames to generate one or more updated data sets.

[0013] Furthermore, according to an exemplary embodiment, the method further comprises: using the one or more updated data sets to train one or more models for control of assisted and automated driving vehicles.

[0014] Furthermore, according to an exemplary embodiment, identifying one or more regions of interest for privacy protection, extracting high-level attributes, generating replacement images, and updating frames are performed via one or more processors disposed on a vehicle acquiring the data.

[0015] Furthermore, according to an exemplary embodiment, identifying one or more regions of interest for privacy protection, extracting high-level attributes, generating replacement images, and updating frames are performed via one or more processors disposed on a vehicle acquiring the data.

[0016] Furthermore, according to an exemplary embodiment, identifying one or more regions of interest for privacy protection, extracting high-level attributes, generating replacement images, and updating frames are performed via one or more processors disposed on one or more road infrastructure systems that obtain data.

[0017] In another exemplary embodiment, a system is provided, which includes one or more cameras and one or more processors. The one or more cameras are configured to obtain data including one or more frames. The one or more processors are connected to the one or more cameras and are configured to at least facilitate: identifying one or more regions of interest for privacy protection for each of the one or more frames; for each of the one or more regions of interest, extracting high-level properties of the region of interest of the frame; generating a replacement image for each of the one or more regions of interest, thereby protecting privacy while maintaining the high-level properties of the region of interest; and for each of the one or more frames, updating the frame with the replacement image of each of the one or more regions of interest, generating an updated frame that protects individual privacy.

[0018] Furthermore in accordance with an exemplary embodiment, the one or more regions of interest relate to a license plate of the vehicle; and the replacement image includes a composite license plate image having high level attributes of the license plate.

[0019] In addition, according to an exemplary embodiment, the one or more processors are further configured to at least facilitate: comparing the replacement image with a plurality of actual license plate images from other vehicles; and utilizing the replacement image only when a statistically distributed match is determined between the replacement image and the plurality of actual license plate images from other vehicles.

[0020] Further in accordance with an exemplary embodiment, the one or more regions of interest relate to a person's face; and the replacement image includes a synthesized human facial image having high-level attributes of the person's face.

[0021] In addition, according to an exemplary embodiment, the one or more processors are further configured to at least facilitate: comparing the replacement image with a plurality of actual facial images from other persons; and utilizing the replacement image only when a match in statistical distribution is determined between the replacement image and the plurality of actual facial images from other persons.

[0022] In addition, according to an exemplary embodiment, one or more cameras are configured to obtain data including video camera images of vehicles and people along one or more roads on which the vehicle is traveling; and the one or more processors are also configured to at least facilitate: updating one or more data sets in a computer memory using updated frames of each of the one or more frames to generate one or more updated data sets; and using the one or more updated data sets to train one or more models for control of assisted and autonomous driving vehicles.

[0023] In another exemplary embodiment, a system is provided, which includes one or more cameras and one or more processors. The one or more cameras are configured to obtain observation data, which includes one or more frames of multiple participants (including vehicles and people) along one or more roads traveling by vehicles. One or more processors are connected to the one or more cameras and are configured to at least facilitate: identifying one or more regions of interest for privacy protection related to vehicles and people for each of the one or more frames; for each of the one or more regions of interest, extracting high-level properties of the region of interest of the frame; generating a synthetic replacement image for each of the one or more regions of interest, thereby protecting privacy while maintaining the high-level properties of the region of interest; for each of the one or more frames, updating the frame with a synthetic replacement image of each of the one or more regions of interest, generating an updated frame that protects individual privacy; using the updated frame of each of the one or more frames to update one or more data sets in a computer memory, generating one or more updated data sets; using the one or more updated data sets to train one or more models for the control of assisted and autonomous vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present disclosure will hereinafter be described in conjunction with the following drawings, wherein like reference numerals refer to like elements, and wherein:

[0025] Figure 1 is a functional block diagram of a system including a vehicle, a remote server, and an infrastructure, according to an exemplary embodiment, the system being configured to generate data including personal information and that can be used in conjunction with autonomous vehicle training and / or other contexts;

[0026] Figure 2 is a method for generating data including personal information according to an exemplary embodiment and can be used with Figure 1 A flowchart of the process of using the system in conjunction with the

[0027] Figure 3 According to an exemplary embodiment Figure 2 A flowchart of a sub-process of a process, that is, the sub-process is used to generate data including a license plate of a vehicle and can be used with Figure 1 Used in combination with the system;

[0028] Figure 4 According to an exemplary embodiment Figure 2 A flowchart of a sub-process of a process of, that is, the sub-process is used to generate data including individual faces and can be used with Figure 1 Used in combination with the system;

[0029] Figure 5A , Figure 5B and Figure 5C (also collectively referred to herein as FIG. 5 ) provides a schematic diagram according to an exemplary embodiment, including Figure 4 of the subprocess Figure 2 An exemplary implementation of the process of; and

[0030] Fig. 6A , Figure 6B and Figure 6C (also collectively referred to herein as FIG. 6 ) provides a schematic diagram according to an exemplary embodiment, including Figure 3 of the subprocess Figure 2 Another exemplary implementation of the process. DETAILED DESCRIPTION

[0031] The following detailed description is merely exemplary in nature and is not intended to limit the present disclosure or its application and uses. Furthermore, there is no intention to be bound by any theory presented in the preceding background or the following detailed description.

[0032] Figure 1 The system 10 is illustrated. In various embodiments, and as described below, the system 10 includes one or more vehicles 100, a remote server 170, and an infrastructure 180. In various embodiments, the system 10 provides for the generation of data, including observational data associated with personal information (such as a person's face and a vehicle license plate number), in a manner that protects the privacy of the personal information while also maintaining maximum use of the data. In various embodiments, according to an exemplary embodiment, the system 10 provides for the generation of data, including observational data associated with personal information (such as a person's face and a vehicle license plate number). Figure 2 The process 200 and Figure 3 The sub-processes and implementations of Figure 6 perform these tasks.

[0033] Specifically, as described in further detail below, in various embodiments, the vehicle 100 and infrastructure 180 collect observational data that includes personal information, such as drawings of individual human faces and / or license plate numbers and information of individual vehicles (e.g., vehicle 100 and / or other vehicles), etc. Also in various embodiments and as described in further detail below, the system 10 processes the observational data in a manner that protects the personal information (and associated privacy) but also maintains optimal efficacy of the data. For example, as described below in conjunction with Figure 1 as well as Figure 2 As further described in more detail in FIG. 6 , system 10 replaces personal information (including face and license plate number) with synthetic replacement information that protects the privacy of the individual but also includes a high level of detail of the information (thereby providing more value to the data than simply masking or obscuring the personal information, etc.). In various embodiments, via Figure 1Processing data in this manner may be performed by one or more processors depicted in FIG. 1 , such as processor 142 of vehicle 100 , processor 184 of infrastructure 180 , and / or processor 173 of remote server 170 .

[0034] In various embodiments, vehicle 100 represents one of a number of different vehicles 100 operating on a roadway or other path (collectively referred to herein as a "road"). Figure 1 A single vehicle 100 is depicted in the figure, but it should be understood that in certain embodiments, the system 10 may include any number of vehicles 100 working together with the remote server 170 and the infrastructure 180, which together perform the following further described in conjunction therewith, Figure 2 The process 200 depicted in Figure 3 -6 sub-processes and implementations. In addition, although the singular term "vehicle" may sometimes be used, it should be understood that this refers to any number of different vehicles (e.g., in a formation or otherwise used together in the system 10 and in performing Figure 2 The process 200 and Figure 3 to the sub-process and implementation of Figure 6).

[0035] In various embodiments, each vehicle 100 comprises an automobile. In some embodiments, the vehicle 100 may be any of many different types of automobiles such as a sedan, van, truck, or sport utility vehicle (SUV), and may be two-wheel drive (2WD) (i.e., rear-wheel drive or front-wheel drive), four-wheel drive (4WD), or all-wheel drive (AWD), and / or various other types of vehicles. In some embodiments, the vehicle 100 may also include a motorcycle or other vehicle, such as an aircraft, a spacecraft, a ship, etc., and / or one or more other types of mobile platforms (e.g., a robot and / or other mobile platforms).

[0036] In some embodiments, some vehicles 100 (in the formation) may be operated in whole or in part by a human driver, while other vehicles 100 may include autonomous or semi-autonomous vehicles, for example where vehicle control (including acceleration, deceleration, braking, and / or steering) is planned and implemented automatically in whole or in part by the control system 102. In addition, in some embodiments, some vehicles 100 may be operated by humans at certain times and via automatic control at other times. Furthermore, in various embodiments, some vehicles 100 include automatic functions via a computer model that is trained using data generated and processed via the system 10 after protecting its privacy.

[0037] In the depicted embodiment, the vehicle 100 includes a body 104 disposed on a chassis 116. The body 104 substantially surrounds the other components of the vehicle 100. The body 104 and the chassis 116 can together form a frame. The vehicle 100 also includes a plurality of wheels 112. The wheels 112 are each rotatably coupled to the chassis 116 near a respective corner of the body 104 to facilitate movement of the vehicle 100. In one embodiment, the vehicle 100 includes four wheels 112, although this may be different in other embodiments (e.g., for trucks and certain other vehicles).

[0038] The drive system 110 is mounted on the chassis 116 and drives the wheels 112, for example via the shaft 114. The drive system 110 preferably includes a propulsion system. In certain exemplary embodiments, the drive system 110 includes an internal combustion engine and / or an electric motor / generator coupled to its transmission. In certain embodiments, the drive system 110 may be different, and / or two or more drive systems 110 may be used. As an example, the vehicle 100 may also incorporate any one or a combination of many different types of propulsion systems such as the following: for example, a combustion engine of gasoline or diesel fuel, a "flexible fuel vehicle" (FFV) engine (i.e., using a mixture of gasoline and alcohol), an engine of a gaseous compound (e.g., hydrogen and / or natural gas) fuel, a combustion / electric motor hybrid engine, and an electric motor.

[0039] As mentioned above, in some embodiments, the vehicle 100 includes one or more functions that can be automatically controlled via the control system 102, for example, in certain circumstances. In some embodiments, some vehicles 100 can be operated by a human driver, while other vehicles 100 can be assisted and automated vehicles, such as semi-autonomous vehicles or fully autonomous vehicles, for automatic control of the drive system 110 and / or other vehicle components. However, this can be different in other embodiments, such as where a human driver can control the drive system 110.

[0040] like Figure 1As depicted, in various embodiments, vehicle 100 also includes a braking system 106 and a steering system 108. In an exemplary embodiment, braking system 106 controls braking of vehicle 100 using braking components controlled via input provided by the driver (e.g., via a brake pedal in certain embodiments) and / or automatically controlled via control system 102. Also in an exemplary embodiment, steering system 108 controls steering of vehicle 100 via steering components (e.g., steering wheel 109 as part of a steering column coupled to shaft 114 and / or wheels 112) controlled via input provided by the driver (e.g., via a steering wheel 109 in certain embodiments) and / or automatically controlled via control system 102.

[0041] exist Figure 1 In the depicted embodiment, in certain embodiments, the control system 102 is coupled to a braking system 106, a steering system 108, and a drive system 110. In various embodiments, the control system 102 facilitates at least generating and processing observation data of the vehicle 100 and / or other vehicles. Additionally, in certain embodiments where the vehicle 100 is an autonomous or semi-autonomous vehicle, the control system 102 also provides control of automated features of the vehicle 100 (including automated operation of the braking system 106, the steering system 108, and / or the drive system 110) in certain circumstances, including using one or more models that are trained using the observation data.

[0042] like Figure 1 As depicted, in various embodiments, control system 102 includes a sensor array 120 , a display 124 , a transceiver 126 , and a controller 140 .

[0043] In various embodiments, the sensor array 120 obtains sensor data for generating observation data. In various embodiments, the sensor array 120 includes one or more cameras 130 (such as video cameras and / or in some embodiments, still image cameras). In addition, in some embodiments, the sensor array 120 may also include one or more other detection sensors 132 (e.g., radar, sonar, lidar, etc.) and / or other sensors 134 (e.g., vehicle position sensors, speed sensors, accelerometers, brake sensors, steering sensors, etc.).

[0044] In various embodiments, the camera 130 obtains observation data of the vehicle 100. In various embodiments, the observation data relates to the road on which the vehicle 100 is operating and its activities and participants, such as the location and movement of pedestrians, vehicles, drivers, and various other details on the road and activities related thereto. In various embodiments, video camera images are obtained. In certain embodiments, static camera images may also be obtained. In various embodiments, the observation data includes personal information, such as the face of a human individual and the license plate number of an individual vehicle (and, for example, may be protected by applicable laws, regulations and / or ordinances).

[0045] In various embodiments, detection sensors 132 and / or other sensors 134 obtain additional information about the road and / or the operation of the vehicle 100 itself (e.g., its position, speed, acceleration, etc.) for use in operating the vehicle 100, for example, based on autonomous operation of the vehicle 100 and / or certain components thereof.

[0046] In certain embodiments, vehicle 100 also includes a transceiver 126. In various embodiments, transceiver 126 communicates with a remote server 170 and / or infrastructure 180 related to observation data and / or processing thereof.

[0047] In certain embodiments, the display 124 provides information to the driver and / or other passengers of the vehicle 100 , such as related to observations and / or operation of the vehicle 100 .

[0048] In various embodiments, controller 140 is coupled to sensor array 120 as well as braking system 106 , steering system 108 , and drive system 110 . Controller 140 is also coupled to display 124 and transceiver 126 in various embodiments.

[0049] In various embodiments, the controller 140 comprises a computer system and includes a processor 142, a memory 144, an interface 146, a storage device 148, and a computer bus 149. In various embodiments, the controller (or computer system) 140 obtains sensor data from the sensor array 120 and, in some embodiments, obtains additional data via the transceiver 126. In various embodiments, the controller 140 processes the observation data, including protecting privacy while maintaining maximum efficacy of the observation data. In some embodiments, the controller 140 also uses the observation data to develop, train, and / or implement one or more autonomous driving models of the vehicle 100 (e.g., for automatic control of the braking system 106, the steering system 108, and / or the drive system 110, and / or one or more related features, such as blind spot or pedestrian detection, etc.). In various embodiments, the controller 140 generates, trains, and / or implements autonomous driving models of the vehicle 100 based on the observation data. Figure 2These and other functionalities are provided by the steps of the process and implementation depicted in FIG. 6 and described further below in conjunction therewith.

[0050] In various embodiments, the controller 140 (and, in some embodiments, the control system 102 itself) is disposed within the body 104 of the vehicle 100. In one embodiment, the control system 102 is mounted on the chassis 116. In some embodiments, the controller 140 and / or the control system 102 and / or one or more components thereof may be disposed external to the body 104, such as on a remote server, in the cloud, or other device that remotely performs image processing. In some embodiments, the controller 140 of the vehicle 100 also performs functions in coordination with a remote server 170 and / or infrastructure 180, as further described below.

[0051] It should be understood that the controller 140 may be different from Figure 1 For example, controller 140 may be coupled to or may otherwise utilize one or more remote computer systems and / or other control systems, such as as part of one or more of the vehicle 100 devices and systems described above.

[0052] In the depicted embodiment, the computer system of controller 140 includes a processor 142, a memory 144, an interface 146, a storage device 148, and a bus 149. Processor 142 performs computational and control functions of controller 140 and may include any type of processor or multiple processors, a single integrated circuit such as a microprocessor, or any suitable number of integrated circuit devices and / or circuit boards that work in conjunction to implement the functionality of a processing unit. During operation, processor 142 executes one or more programs 150 contained in memory 144 and, therefore, controls the overall operation of controller 140 and the computer system of controller 140, generally in performing the processes described herein, such as Figure 2 - The process and implementation depicted in FIG. 6 and as further described below in conjunction therewith.

[0053] Memory 144 may be any type of suitable memory. For example, memory 144 may include various types of dynamic random access memory (DRAM) such as SDRAM, various types of static RAM (SRAM), and various types of non-volatile memory (PROM, EPROM, and flash memory). In some examples, memory 144 is located on the same computer chip and / or co-located on the same computer chip as processor 142. In the depicted embodiment, memory 144 stores the above-described program 150 as well as one or more databases 155 (e.g., related to observation data) and other stored values ​​156 (e.g., including as described in Figure 2to the threshold for processing observation data as set forth in the process and implementation depicted in FIG. 6 and as further described below in conjunction therewith according to an exemplary embodiment).

[0054] The bus 149 is used to transfer programs, data, status and other information or signals between the various components of the computer system of the controller 140. The interface 146 allows communication with the computer system of the controller 140, such as from a system drive and / or another computer system, and can be implemented using any suitable method and device. In one embodiment, the interface 146 obtains various data from the sensor array 120 and / or the navigation system 122. The interface 146 may include one or more network interfaces to communicate with other systems or components. The interface 146 may also include one or more network interfaces to communicate with technicians, and / or one or more storage interfaces to connect to storage devices, such as storage device 148.

[0055] Storage device 148 may be any suitable type of storage device, including various types of direct access storage and / or other memory devices. In an exemplary embodiment, storage device 148 comprises a program product from which memory 144 may receive program 150, which executes Figure 2 In another exemplary embodiment, the program product may be directly stored in and / or otherwise accessed by memory 144 and / or secondary storage devices (e.g., disk 157), such as those mentioned below.

[0056] The bus 149 may be any suitable physical or logical means of connecting computer systems and components. This includes, but is not limited to, direct hard-wired connections, fiber optics, infrared, and wireless bus technologies. During operation, the program 150 is stored in the memory 144 and executed by the processor 142.

[0057] It should be understood that while this exemplary embodiment is described in the context of a fully functional computer system, those skilled in the art will recognize that the mechanisms of the present disclosure can be distributed as a program product and that one or more types of non-transitory computer-readable signal-bearing media are used to store the program and its instructions and implement its distribution, such as a non-transitory computer-readable medium that carries the program and contains computer instructions stored therein for causing a computer processor (such as, processor 142) to execute and implement the program. Such a program product can take a variety of forms, and the present disclosure applies equally regardless of the specific type of computer-readable signal-bearing medium used to implement the distribution. Examples of signal-bearing media include: recordable media, such as floppy disks, hard drives, memory cards, and optical disks; and transmission media, such as digital and analog communication links. It should be understood that cloud-based storage and / or other technologies may also be utilized in certain embodiments. Similarly, it should be understood that the computer system of controller 140 may also differ in other ways. Figure 1 In the embodiments depicted in FIG. 1 , a computer system such as controller 140 may be coupled to or may otherwise utilize one or more remote computer systems and / or other control systems.

[0058] Continue to refer to Figure 1 In various embodiments, the vehicle 100, the remote server 170, and the infrastructure 180 communicate via one or more communication networks 160. In various embodiments, the communication network 160 may include one or more wireless communication networks (e.g., satellite-based, cellular, and / or any number of other different types of wireless communication networks).

[0059] Furthermore, in various embodiments, the remote server 170 is disposed remotely or at a significant physical distance from each of the infrastructure 180 and the vehicles 100 (e.g., in a formation). Figure 1 As depicted in FIG. 1 , remote server 170 includes one or more transceivers 172, processor 173, and computer memory 174 storing a map database 176 and stored values ​​178. In various embodiments, transceiver 172 is used to communicate with vehicle 100 and infrastructure 180, including regarding observation data and its processing. Figure 1As depicted, transceiver 172, processor 173, memory 174, database 176, and stored values ​​178 are similar or identical to corresponding features of vehicle 100 (e.g., with respect to its transceiver 126, processor 142, memory 144, database 155, and stored values ​​156). In addition, in certain embodiments, processor 173 processes or facilitates the processing of observation data from vehicle 100 (and / or from infrastructure 180), including for protecting privacy while maintaining maximum efficacy of the observation data (e.g., as described below in conjunction with Figure 2 6 and the implementation thereof are further described).

[0060] In addition, in various embodiments, the infrastructure 180 is disposed remotely from the vehicle 100 and the remote server 170. In various embodiments, the infrastructure 180 includes one or more smart traffic lights, smart traffic signs, etc., as well as other possible devices and / or systems that generate observation data containing personal information (such as a person's face and a vehicle license plate number). Figure 1 As depicted, infrastructure 180 includes one or more sensors 181, including one or more cameras 182 for obtaining observation data (e.g., similar to sensor array 120 including camera 130 of vehicle 100). In addition, in various embodiments, infrastructure 180 also includes one or more transceivers 183, processor 184, and computer memory 185 storing database 187 and stored values ​​188. In various embodiments, transceiver 183 is used to communicate with vehicle 100 and remote server 170, including regarding observation data and its processing. Figure 1 As depicted, transceiver 183, processor 184, memory 185, database 187, and stored values ​​188 are similar or identical to corresponding features of vehicle 100 (e.g., with respect to its transceiver 126, processor 142, memory 144, database 155, and stored values ​​156). In addition, in certain embodiments, processor 184 processes or facilitates the processing of observation data from vehicle 100 (and / or from infrastructure 180 itself), including for protecting privacy while maintaining maximum efficacy of the observation data (e.g., as described below in conjunction with Figure 2 6 and the implementation thereof are further described).

[0061] Figure 2 is a method for generating data including personal information according to an exemplary embodiment and can be used with Figure 1 Flow chart of process 200 for use in conjunction with system 10. Figure 3 and Figure 4 5 and 6 (which depict exemplary implementations of process 200) describe in more detail Figure 2 process 200.

[0062] Continue to refer to Figure 2 In various embodiments, process 200 begins at step 202. In various embodiments, when (e.g., by Figure 1 Process 200 begins when a vehicle 100 and / or infrastructure 180) is collecting or is about to collect observation data.

[0063] In various embodiments, such as Figure 2 As shown, the data image is processed to protect privacy along the first path 206, the second path 208, or both. In various embodiments, the first path 206 involves protection of vehicle license plate information starting from step 210. In various embodiments, the first path 206 is also represented as being associated with the first path 206. Figure 3 In various embodiments, the second path 208 involves the protection of the facial information of the person starting from step 216. In various embodiments, the second path 208 is also represented as associated with the second path 208. Figure 4 The sub-process in and the related implementation method of Figure 5.

[0064] Continue to refer to Figure 2 In an exemplary embodiment, during step 210 of the first path 206, a camera video frame is obtained and identified. In various embodiments, the camera video frame includes an image obtained from one or more cameras (such as camera 130 of vehicle 100 and / or camera 182 of infrastructure 180) including observation data of one or more vehicles. Figure 1 One or more processors analyze each camera video frame individually.

[0065] In addition, in various embodiments, in step 212, for each camera video frame, the vehicle license plate in the camera video frame is identified. In various embodiments, this is done using observation data via Figure 1 One or more processors to execute.

[0066] Furthermore, in various embodiments, the selected attribute is extracted from the video frame (step 214). Figure 1 One or more processors extract certain selected high-level or abstract attributes about the license plate from the video frame. In addition, in some embodiments, the extracted features include fonts, colors, styles, and jurisdictions and / or authorities associated with the license plate.

[0067] Furthermore, in various embodiments, and referring now to the second path 208, in an exemplary embodiment, in various embodiments during step 216 of the second path 208, a camera video frame is obtained and identified. In various embodiments, the camera video frame includes an image including observation data of one or more persons obtained from one or more cameras, such as the camera 130 of the vehicle 100 and / or the camera 182 of the infrastructure 180. In various embodiments, the camera video frame is obtained via Figure 1 One or more processors analyze each camera video frame individually.

[0068] Additionally, in various embodiments, during step 218, for each camera video frame, a human face in the camera video frame is identified. In various embodiments, this is done using observation data via Figure 1 One or more processors to execute.

[0069] Furthermore, in various embodiments, the selected attribute is extracted from the video frame (step 220). Figure 1 One or more processors extract some selected high-level or abstract attributes about human face from video frames. In addition, in some embodiments, the extracted features include estimated age, race, size, gender, etc. of human face.

[0070] In various embodiments, further processing is performed on the images of both paths (step 222). Specifically, in various embodiments, during step 222, generative artificial intelligence processing is applied to the images, and in particular to vehicle license plates and human faces detected in the video camera frame. In certain embodiments, a generative artificial intelligence (AI) module is applied to the observed data for license plates and human faces via a text-to-image generator. In various embodiments, this is via Figure 1 One or more processors to execute.

[0071] In various embodiments, a privacy-preserving lossless output is generated in steps 224 and 226 as a result of generative artificial intelligence processing applied to both license plates and human faces.

[0072] Specifically, in various embodiments, during step 224 (as part of first path 206), via Figure 1 One or more processors generate a privacy-preserving lossless output that replaces an actual license plate in a camera video frame with a synthetic replacement license plate, where the synthetic replacement license plate is generated to have a different license plate number than the actual license plate but to have other high-level characteristics similar to the actual license plate (e.g., having the same or similar font, color, style, and jurisdiction and / or authority associated with the license plate).

[0073] Additionally, also in various embodiments, during step 226 (as part of the second path 208), via Figure 1 One or more processors generate a privacy-preserving lossless output that replaces an actual human face in a camera video frame with a replacement “human” face, where the replacement “human” face is generated to have a different actual human face than the actual human face in the image, but has similar other high-level features as the actual human face in the image (e.g., the same or similar estimated age, race, size, gender, etc. as the human face in the image).

[0074] In various embodiments, one or more data sets are updated (step 228). Specifically, in various embodiments, during step 228, one or more data sets are obtained for the observed data, with privacy protection associated with the outputs of steps 224 and 226. Specifically, in various embodiments, the updated data set of step 228 includes the observed data of steps 210, 216, but replaces the license plate number, human face and / or other personal / private information with the synthesized output of steps 224 and 226 to include corresponding high-level features while maintaining individual privacy and the validity of the data set. In various embodiments, the updated data set of step 228 is presented in the database 155 of the vehicle 100, the database 178 of the remote server 170, and / or the database 187 of the infrastructure 180.

[0075] In various embodiments, the updated data is implemented (step 230). In various embodiments, the updated data set of step 228 is used to develop and train models for assisted and autonomous vehicles (such as vehicle 100 and / or other vehicles). Specifically, in some embodiments, Figure 1 One or more processors utilize Figure 1 The updated observation data of the updated dataset is used to develop and train one or more autonomous vehicle models for operation of assisted and automated driving vehicles using machine learning, such as for automatic control of the braking system 106, the steering system 108 and / or the drive system 110, and / or one or more related features (such as blind spot or pedestrian detection, etc.).

[0076] In various embodiments, the process then terminates at step 232 .

[0077] Figure 3 According to an exemplary embodiment Figure 2 The process 200 corresponds to Figure 2 Flow chart of a sub-process of the first path 206, that is, the sub-process is used to generate data including a license plate of a vehicle and can be used with Figure 1 The system 10 is used in combination. This article also describes FIG. Figure 36 depicts an exemplary implementation of process 200 in which the observation data relates to a license plate of a vehicle according to an exemplary embodiment.

[0078] like Figure 3 As shown, in various embodiments, Figure 1 During the first path 206 of the process 200, when the Figure 1 The process starts at step 301 when the camera video frame analysis data image of step 210 is performed.

[0079] In various embodiments, for each camera video frame, Figure 3 At step 302, the vehicle license plate in the camera video frame is identified. In various embodiments, this is done using observation data via Figure 1 Specifically, refer to Fig. 6A In various embodiments, during step 302 , a license plate 603 of a vehicle 602 , as well as a body 604 of the vehicle 602 and / or other features thereof are detected for a particular image 601 .

[0080] Return to reference Figure 3 In various embodiments, mask detection and segmentation is provided (step 304). In various embodiments, this is done using observation data via Figure 1 Specifically, refer to Figure 6B In various embodiments, during step 304, a region of interest 611 (eg, a rectangular box) is generated for the license plate 603 in the first updated image 610. Figure 6B As depicted, in various embodiments, the region of interest 611 includes the license plate 603 , but does not include the body 604 or additional components of the vehicle 602 .

[0081] Return to reference Figure 3 , in various embodiments, additional processing is performed, such as Figure 3 The combined step (or sub-process) 305 is shown in FIG. In various embodiments, the various steps (described below) of the combined step 305 are Figure 1 One or more processors execute.

[0082] Specifically, in various embodiments, as part of the combining step 305, abstract attributes are extracted from the image (step 306). Specifically, similar to the above discussion, in various embodiments, via Figure 1 One or more processors extract certain selected high-level or abstract attributes about the license plate from the video frame. In addition, in some embodiments, the extracted features include fonts, colors, styles, and jurisdictions and / or authorities related to the license plate.

[0083] In various embodiments, a license plate generator is used to generate an updated license plate of the image (step 308). Specifically, in various embodiments, Figure 1 One or more processors of the system generate a new, fictitious, composite license plate for the image that includes high-level abstract features of the original license plate, but without the personal information of the original license plate (e.g., license plate number). In various embodiments, this is performed using generative artificial intelligence, for example, as described above in conjunction with Figure 2 as described in step 222.

[0084] Additionally, in various embodiments, during step 312, from one or more databases 310 (such as Figure 1 The database shown in FIG. 300 is used to obtain actual license plate images from other vehicles (other than vehicle 100) and compare them with the updated license plate generated from step 308. In various embodiments, these steps are performed by Figure 1 is executed by one or more of the processors.

[0085] In various embodiments, a determination is made as to whether a match exists (step 316). Specifically, in various embodiments, a determination is made as to whether a match exists between the license plate generated in step 308 and the actual license plate from the other vehicle in step 312. In certain embodiments, a match is determined when the difference between the license plate in step 308 and the actual license plate from the other vehicle in step 312 is less than a predetermined calibratable threshold 314 (e.g., as a threshold Figure 1 In addition, in some embodiments, via one or more processors, a determination as to whether a match exists is performed using a predetermined calibratable threshold 314 using a Gaussian distribution (including the mean and variance of pixels and / or other image attributes).

[0086] In various embodiments, if it is determined during step 316 that there is no match, the process returns to step 308, such as generating a new updated license plate for the image. In various embodiments, steps 308-316 are repeated thereafter until it is determined during an iteration of step 316 that there is a match.

[0087] In various embodiments, once a match is determined during an iteration of step 316, process 200 proceeds to step 318. In various embodiments, during step 318, a privacy-preserving output is generated. Figure 2Step 224 of the description generates, as a result of applying the generative artificial intelligence processing to the license plate, a privacy-preserving lossless output that replaces the actual license plate of the camera video frame with a replacement synthetic license plate, the replacement synthetic license plate being generated to have a license plate number that is different from the actual license plate, but having other high-level features that are similar to the actual license plate (e.g., having the same or similar font, color, style, and jurisdiction and / or authority associated with the license plate). For example, reference Figure 6C In various embodiments, a new, updated license plate 623 is provided in the second updated image 620 (without any changes to the body 604 or other components of the vehicle 602).

[0088] In various embodiments, one or more data sets are updated (step 320). Specifically, similar to Figure 2 In various embodiments, during step 320, one or more data sets are obtained for the observed data, but with the privacy protections associated with the output of step 318. Specifically, in various embodiments, the updated data set of step 320 includes the observed data, but with the license plate number and / or other personal / private information replaced with the newly generated license plate of step 308. In various embodiments, the updated data set of step 320 is Figure 3 In various embodiments, the updated database 330 may be presented in Figure 1 in the database 155 of the vehicle 100 , the database 178 of the remote server 170 , and / or the database 187 of the infrastructure 180 .

[0089] In various embodiments, the updated data is implemented (step 322). Similar to Figure 2 In step 230, in various embodiments, the updated data set is used to develop and train a model for assisted and autonomous driving of vehicles (such as vehicle 100 and / or other vehicles). Specifically, in some embodiments, Figure 1 One or more processors utilize Figure 1 The updated observation data of the updated dataset is used to develop and train one or more autonomous vehicle models for operation of assisted and automated driving vehicles using machine learning, such as for automatic control of the braking system 106, the steering system 108 and / or the drive system 110, and / or one or more related features (such as blind spot or pedestrian detection, etc.).

[0090] In various embodiments, the process then terminates at step 324 .

[0091] Now refer to Figure 4 , according to an exemplary embodiment, Figure 2 The process of 200 Figure 2 Flow chart of the sub-process corresponding to the second path 208 of the embodiment, that is, the sub-process is used to generate data including a human face and can be used with Figure 1 The system 10 is used in combination. This article also describes FIG. Figure 4 5 depicts an exemplary implementation of process 200 according to an exemplary embodiment, wherein the observed data relates to a human face.

[0092] like Figure 4 As shown, in various embodiments, Figure 1 During the second path 208 of the process 200, when the Figure 1 The process begins at step 401 when the camera video frame analysis data image of step 210 is performed.

[0093] In various embodiments, for each camera video frame, Figure 4 At step 402, a face recognition is performed on a human face in a camera video frame. In various embodiments, this is performed using observation data via Figure 1 Specifically, refer to Figure 5A In various embodiments, during step 402 , faces 503 , 506 of respective persons 502 , 505 , as well as bodies 504 , 507 and / or other features thereof, of respective persons 502 , 505 are detected for a particular image 501 .

[0094] Return to reference Figure 4 In various embodiments, mask detection and segmentation is provided (step 404). In various embodiments, this is done using observation data via Figure 1 Specifically, refer to Figure 5B In various embodiments, during step 404, regions of interest 511, 512 (eg, rectangular boxes) are generated for the faces 503, 506 of the respective persons 502, 505 in the first update image 510. Figure 5B As depicted, in various embodiments, the regions of interest 511 , 512 include the faces 503 , 506 , but exclude the bodies 504 , 507 of the respective persons 502 , 505 .

[0095] Return to reference Figure 4 , in various embodiments, additional processing is performed, such as Figure 4 The combined step (or sub-process) 405 is shown in FIG. In various embodiments, the various steps (described below) of the combined step 405 are represented by Figure 1 One or more processors execute.

[0096] Specifically, in various embodiments, as part of the combining step 405, abstract attributes are extracted from the image (step 406). Specifically, similar to the above discussion, in various embodiments, via Figure 1 One or more processors extract some selected high-level or abstract attributes about human faces from video frames. In addition, in some embodiments, the extracted features include age, race, size and gender associated with the licensed human face.

[0097] In various embodiments, a human face generator is used to generate an updated face (or faces) of the image (step 408). In some embodiments, the human face generator may be referred to as a vulnerable road user (VRU) face generator, such as for human pedestrians on the road. Specifically, in various embodiments, Figure 1 The one or more processors of the system can generate a new, fictitious, synthetic facial image of one or more people for the image, which includes high-level abstract features of the person's face, but in a way that is not identifiable to a specific face, thereby protecting the privacy of the person. In various embodiments, this is performed using generative artificial intelligence, for example, as described above in conjunction with Figure 2 as described in step 222.

[0098] Additionally, in various embodiments, during step 412, from one or more databases 410 (such as Figure 1 The actual human face image from other people is obtained from the database shown in FIG404 and compared with the human face generated from step 408. In various embodiments, these steps are performed by Figure 1 One or more processors for execution.

[0099] In various embodiments, it is determined whether there is a match (step 416). Specifically, in various embodiments, it is determined whether there is a match between the face of the person in step 408 and the actual face of the person from the other person in step 412. In some embodiments, when the difference between the face of the person in step 408 and the face of the person from the other person in step 412 is less than a predetermined calibrable threshold 414 (e.g., as Figure 1 In addition, in some embodiments, via one or more processors, a determination as to whether a match exists is performed using a predetermined calibratable threshold 414 using a Gaussian distribution (including the mean and variance of pixels and / or other image attributes).

[0100] In various embodiments, if it is determined during step 416 that there is no match, the process returns to step 408, such as generating a new updated human face for the image. In various embodiments, steps 408-416 are repeated thereafter until it is determined during an iteration of step 416 that there is a match.

[0101] In various embodiments, once a match is determined during an iteration of step 416, process 200 proceeds to step 418. In various embodiments, during step 418, a privacy-preserving output is generated. Figure 2 Step 226 of the description generates, as a result of applying the generative artificial intelligence processing to the human face, a privacy-preserving lossless output that replaces the actual human face of the camera video frame with a replacement synthetic human face, the replacement synthetic human face being generated to have a different face that is not recognizable from the actual human face (in order to protect privacy), but having other high-level features similar to the actual human face (e.g., having the same or similar age, race, size, and / or gender as the original human face). For example, referring to Figure 5C In various embodiments, new, updated human faces 523, 526 are provided in the second updated image 520 (without any changes to the bodies 504, 507 or other parts of the persons 502, 505).

[0102] In various embodiments, one or more data sets are updated (step 420). Specifically, similar to Figure 2 In various embodiments, during step 420, one or more datasets are obtained for the observed data, but with the privacy protections associated with the output of step 318. Specifically, in various embodiments, the updated dataset of step 420 includes the observed data, but with the faces of the persons replaced with the newly generated faces of step 408. In various embodiments, the updated dataset of step 420 is Figure 4 In various embodiments, the updated database 430 may be presented in Figure 1 in the database 155 of the vehicle 100 , the database 178 of the remote server 170 , and / or the database 187 of the infrastructure 180 .

[0103] In various embodiments, the updated data is implemented (step 422). Similar to Figure 2 In step 230, in various embodiments, the updated data set is used to develop and train a model for assisted and autonomous driving of vehicles (such as vehicle 100 and / or other vehicles). Specifically, in some embodiments, Figure 1 One or more processors utilize Figure 1 The updated observation data of the updated dataset is used to develop and train one or more autonomous vehicle models for assisted and automated vehicle operation using machine learning, such as automatic control of the braking system 106, the steering system 108 and / or the drive system 110, and / or one or more related features (such as blind spot or pedestrian detection, etc.).

[0104] In various embodiments, the process then terminates at step 424 .

[0105] Therefore, methods, systems, and vehicles for generating data sets that protect privacy while still maintaining the best efficacy of the data sets are provided. In various cases, the observation data from the video camera is processed so that personal information such as license plates and human faces are replaced with fictitious license plates, human faces, etc. that protect privacy while maintaining the high-level characteristics of license plates, human faces, etc. In various embodiments, the processed data thereby protects personal privacy while maintaining the best efficacy of the observation data. In various embodiments, the observation data after being processed to maintain privacy is used to develop and train models for assisted and autonomous driving vehicles, such as for automatic control of the braking system 106, the steering system 108, and / or the drive system 110, and / or one or more related features, such as blind spot or pedestrian detection, etc.

[0106] In various embodiments, detecting and protecting privacy for license plate numbers and human faces allows for maximum efficacy while maintaining privacy and complying with applicable government, agency and / or other laws, regulations and / or requirements. In various embodiments, other personal information other than license plate information and human faces and / or in addition to license plate information and human faces can also be processed in a similar manner through the processes and implementations described herein to protect privacy while also maintaining the efficacy of the observed data.

[0107] It should be understood that the systems, vehicles, and methods may vary from those depicted in the figures and described herein. Figure 1 The vehicle 100, the remote server 170 and the infrastructure and its components Figure 1 The system 10 can be used with Figure 1 Similarly, it should be understood that Figure 2 - The steps of the process and implementation of FIG. 6 may be different than depicted in the figures, and / or various steps may occur simultaneously and / or in a different order than depicted in the figures.

[0108] Although at least one exemplary embodiment has been presented in the foregoing detailed description, it should be understood that there are a large number of variations. It should also be understood that the exemplary embodiment or multiple exemplary embodiments are merely examples and are not intended to limit the scope, applicability, or configuration of the present disclosure in any way. On the contrary, the foregoing detailed description will provide a convenient roadmap for those skilled in the art to implement the exemplary embodiment or multiple exemplary embodiments. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the scope of the present disclosure as set forth in the attached claims and their legal equivalents.

Claims

1. A method comprising: Obtain data from one or more cameras, the data comprising one or more frames; identifying, via one or more processors, for each of the one or more frames, one or more regions of interest for privacy protection; extracting, via the one or more processors, for each of the one or more regions of interest, high-level properties of the region of interest of the frame; generating, via the one or more processors, a replacement image for each of the one or more regions of interest so as to protect privacy while maintaining high-level attributes of the region of interest; as well as For each of the one or more frames, the frame is updated, via the one or more processors, with the replacement image of each of the one or more regions of interest to generate an updated frame that protects individual privacy.

2. The method according to claim 1, wherein: The one or more areas of interest relate to a vehicle registration plate; and The replacement image includes a composite license plate image having high level attributes of the license plate.

3. The method according to claim 2, wherein: The composite license plate image has the same font, color, style, and jurisdiction as the license plate, but includes a different license plate number.

4. The method according to claim 2, further comprising: comparing, via the one or more processors, the replacement image to a plurality of actual license plate images from other vehicles; Therein, the replacement image is only utilized if a statistically distributed match is determined between the replacement image and the plurality of actual license plate images from the other vehicles.

5. The method according to claim 1, wherein: The one or more regions of interest relate to a person's face; and The replacement image comprises a synthesized human facial image having high level attributes of the person's face.

6. The method according to claim 5, wherein: The synthesized human facial image has the same age, race, and size as the face of the person, but the person cannot be identified from the synthesized human facial image.

7. The method according to claim 5, further comprising: comparing, via the one or more processors, the replacement image to a plurality of actual facial images from other persons; The replacement image is utilized only when a match in statistical distribution is determined between the replacement image and the plurality of actual facial images from the other persons.

8. The method according to claim 1, wherein: The data includes video camera images of vehicles and persons along one or more roads along which the vehicle travels.

9. The method according to claim 1, further comprising: updating one or more data sets in a computer memory using the updated frame of each of the one or more frames to generate one or more updated data sets; as well as The one or more updated datasets are used to train one or more models for assisted and automated vehicle control.

10. A system comprising: one or more cameras configured to obtain data comprising one or more frames; as well as one or more processors coupled to the one or more cameras and configured to at least facilitate: identifying, for each of the one or more frames, one or more regions of interest for privacy protection; For each region of interest of the one or more regions of interest, extracting high-level properties of the region of interest of the frame; generating a replacement image for each of the one or more regions of interest, thereby protecting privacy while maintaining high-level properties of the region of interest; as well as For each of the one or more frames, the frame is updated using the replacement image of each of the one or more regions of interest to generate an updated frame that protects individual privacy.