Modifying Sensor Data Using Generative Adversarial Models

Abstract

A method, system, and apparatus including a computer program encoded on a computer storage medium that uses a generative adversarial model to improve the quality of sensor data generated by a first environmental sensor to be similar to the quality of sensor data generated by another sensor having a higher quality than the first environmental sensor. A first training data set and a second training data set generated by a first environmental sensor having a first quality and a second sensor having a target quality, respectively, are received. The generative adversarial model is trained using the first training data set and the second training data set to modify sensor data from the first environmental sensor by reducing the difference in quality between the sensor data generated by the first environmental sensor and the sensor data generated by the target environmental sensor.

Description

Technical Field

[0001] This specification relates to using a generative adversarial model to improve the quality of sensor data generated by a first environmental sensor to be similar to the quality of sensor data generated by another sensor having a higher quality than the first environmental sensor. Background Art

[0002] Environmental sensors (e.g., audio sensors, video sensors, and image sensors) have different resolutions. As a result, some environmental sensors have a higher resolution than others. For example, image sensors in modern mirrorless cameras typically have a higher resolution than image sensors in point-and-shoot cameras. Devices with higher resolution environmental sensors generally generate higher resolution media compared to devices with lower resolution environmental sensors. Summary of the Invention

[0003] In general, one innovative aspect of the subject matter described in this specification can be embodied in a method that can include the following operations: receiving a first training data set generated by a first environmental sensor having a first quality, the first training data set having the first quality; receiving a second training data set generated by a target environmental sensor having a target quality, the second training data set having the target quality, and the first quality being different from the target quality; using the first training data set and the second training data set to train a generative adversarial model to modify sensor data from the first environmental sensor by reducing the quality difference between the sensor data generated by the first environmental sensor and the sensor data generated by the target environmental sensor, wherein the training includes: obtaining, from a generator model of the generative adversarial model and using one or more data items in the first training data set, a modified sensor data set having a quality different from the first quality; inputting a data item set including one or more data items from the second training sensor data set and the modified first sensor data set into a discriminator model of the generative adversarial model; determining, by the discriminator model and using the data item set, whether each data item in the data item set has the target quality; adjusting the discriminator model and the generator model when the discriminator model determines that a data item in the data item set generated by the generator model has the target quality; and adjusting the discriminator model and the generator model when the discriminator model determines that a data item in the data item set generated by the generator model does not have the target quality. Other embodiments of this aspect include corresponding systems, devices, apparatuses, and computer programs configured to perform the actions of the method. The computer program (e.g., instructions) can be encoded on a computer storage device. These and other embodiments can each optionally include one or more of the following features.

[0004] In some embodiments, each of the first environmental sensor and the second environmental sensor may acquire one of sound, image, or video.

[0005] In some embodiments, the method may include the following operations: receiving first sensor data generated by a first environmental sensor having a first quality; inputting the first sensor data into a generative adversarial model; and using a generator model of the generative adversarial model to obtain modified sensor data based on the input first sensor data.

[0006] In some embodiments, the method may include: receiving first sensor data generated by a first environmental sensor having a first quality; inputting information about the first environmental sensor into a known defect data structure storing known defects of different environmental sensors; obtaining the known defects of the first environmental sensor from the known defect data structure; adjusting the first sensor data based on the known defects of the first environmental sensor; inputting the adjusted first sensor data into a generative adversarial model; and using a generator model of the generative adversarial model to obtain modified sensor data based on the adjusted first sensor data.

[0007] Certain embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. The innovations described in this specification are capable of using low-resolution environmental sensors to generate media having a resolution higher than the resolution of the environmental sensors. Conventional environmental sensors generate sensor data having a resolution that is the same as (or lower than) the resolution of these sensors. Thus, there is a direct correlation between the resolution of an environmental sensor and the resolution of the sensor data acquired by the sensor. The innovations described in this specification use a generative adversarial model (which may also be referred to as a generative adversarial network) to modify sensor data generated by an environmental sensor to be similar to sensor data generated by a different environmental sensor having a higher resolution. As a result, a device having a low-resolution, low-cost sensor can be modified (as described in this specification) to generate high-resolution media that is conventionally generated only by more expensive, higher-resolution sensors. This allows, for example, a user to experience high-resolution media content even though the content was captured with a low-resolution sensor. Additionally, it allows devices having lower-cost and lower-resolution sensors to compete in the market with devices that use more expensive sensors having higher resolutions.

[0008] The innovations described in this specification can be used to generate high-resolution media, even though lower-resolution sensors are used, which are necessary for the environments in which these sensors are deployed. For example, some outdoor applications can only accommodate low-resolution sensors suitable for harsh environments (e.g., outdoors, high turbulence, etc.). As a result, the media generated by the sensors used in these applications typically has lower quality compared to the media generated by higher-resolution sensors. The innovations described in this specification are capable of modifying the sensor data received by the low-resolution sensors to generate high-resolution media similar to that generated by higher-resolution sensors.

[0009] In addition, the innovations described in this specification are capable of using defective sensors to generate high-resolution images. Conventional environmental sensors may have certain defects (present or emerging during the use of the sensors). For example, an image sensor may have defects such as chromatic aberration or stuck pixels, which may occur during the use of these sensors. In such cases, the image sensor can be repaired (which may be expensive due to the complexity of the image sensor), or the image sensor can be replaced with another sensor. The innovations described in this specification help avoid such expensive repairs or replacements of defective environmental sensors. The innovations described in this specification are implemented by using a combination of known defect data structures and generative adversarial models (as further described below) to modify the sensor data generated by the defective sensors to be similar to the sensor data obtained by a normally operating sensor.

[0010] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram of an example environment for training a generative adversarial model.

[0012] Figure 2 is a flowchart of an example process for training a generative adversarial model.

[0013] Figure 3 is a block diagram of an example environment in which a generative adversarial model is used to modify sensor data received from a first environmental sensor.

[0014] Figure 4 is a flowchart of an example process for modifying sensor data received from a first environmental sensor using a generative adversarial model.

[0015] Figure 5 is a block diagram of an example computer system.

[0016] Like reference numerals and names in different figures indicate like elements. DETAILED DESCRIPTION

[0017] This specification relates to using a model, such as a generative adversarial model, to improve the quality of sensor data generated by a first environmental sensor to be similar to the quality of sensor data generated by another sensor having a higher quality than the first environmental sensor. As used in this specification, the quality of sensor data may refer to the measurement aspects of the sensor data, which may include resolution, error rate, fidelity, and signal-to-noise ratio of the sensor data. Although the example embodiments described below use generative adversarial models, other machine learning systems may also be used. Further, although resolution is used as the sensor data quality metric to describe the example embodiments below, other sensor data quality metrics (e.g., error rate, fidelity, and signal-to-noise ratio) may be used to implement the operations and structures described in this specification.

[0018] A generative adversarial model is a neural network model that uses two competing neural network models to generate data having the same characteristics as a training data set. The two competing neural network models are a generator model and a discriminator model. The goal of the generator model is to generate data similar to the training data, and the goal of the discriminator model is to distinguish between real data (i.e., the training data) and fake data (i.e., the generated data).

[0019] The discriminator model and the generator model together attempt to optimize a min-max objective function. The discriminator model attempts to maximize the objective by correctly identifying the training data as real data and the generated data as fake data. On the other hand, the generator model attempts to minimize the objective by generating fake data that is misclassified as real data by the discriminator.

[0020] Conventional generative adversarial models have been trained to generate, for example, images similar to the images in a training data set. In such a model, the generator model of the generative adversarial model uses a random data distribution to generate images. The generated images, along with the training images representing the actual or real data that the generative adversarial model is trying to simulate, are input into the discriminator model. The discriminator model classifies the input images as real (i.e., the images are from the training data set) or fake (i.e., the images are not from the training data set). In some cases, the discriminator model may correctly classify the input images as real, while in other cases, the discriminator model misclassifies the input data as real. In both cases, the classification of the discriminator model (whether correct or incorrect) is used to update the generator model and the discriminator model. This process is repeatedly repeated when training the generative adversarial model.

[0021] As further described in this specification, a novel method described below can be used to train a generative adversarial model to modify sensor data generated by a first environmental sensor having a first quality (e.g., resolution) to be similar to sensor data generated by a target environmental sensor having a target quality (e.g., target resolution). The target quality is different from the first quality and is generally a higher quality than the first quality.

[0022] Figure 1 is a block diagram of an example environment 100 for training such a generative adversarial model.

[0023] Train using two training data sets Figure 1 the generative adversarial model 120 shown. A first training data set is generated using a first environmental sensor having a first resolution. Each data item in the first training data set has the first resolution. A second training data set is generated using a target sensor (different from the first environmental sensor) having a target resolution (different from the first resolution). Each data item in the second training data set has the target resolution. In some embodiments, each data item in the first training data set has a corresponding data item in the second training data set that captures the same environmental stimulus. For example, if the first training data set includes 10 images of 10 different stimuli, the second training data set also includes 10 images of the same stimuli.

[0024] As used in this specification, an environmental sensor is a sensor that acquires a particular type of media (such as, for example, video, audio, or images) in response to an environmental stimulus. Examples of environmental sensors include one or more of the following: image sensors; video sensors; audio sensors; position / location sensors; microelectromechanical systems; motion sensors; accelerometers; magnetometers; and / or gyroscopes. It should be understood that other examples of environmental sensors may also be used.

[0025] The goal is to develop a model that can be deployed with a device using a first sensor and, by using the model, modify the data generated by the first sensor such that the data has the quality of the data that would be generated using a target sensor.

[0026] The following refers to Figure 2 the operation of the generative adversarial model 120 and its components, as shown in the example environment 100.

[0027] Figure 2 is for training Figure 1Flowchart of an example process 200 of the generative adversarial model 120. The operations of process 200 described below are for illustrative purposes only. The operations of process 200 can be performed by any suitable device or system (e.g., any suitable data processing device). The operations of process 200 can also be implemented as instructions stored on a non-transitory computer-readable medium. Execution of the instructions causes one or more data processing devices to perform the operations of process 200.

[0028] Process 200 receives a first set of training data generated by a first environmental sensor having a first resolution (at 202). In some embodiments, process 200 can receive the first set of training data from a content source on a network (e.g., a LAN, WAN, the Internet, or a combination thereof) or from a device including the first environmental sensor. After receiving the first set of training data, process 200 can store the data in the first sensor training data storage device 104.

[0029] Process 200 receives a second set of training data generated by a target sensor having a target resolution (at 204). In some embodiments, process 200 can receive the second set of training data from a content source on a network (e.g., a LAN, WAN, the Internet, or a combination thereof) or from a device including the target environmental sensor. After receiving the second set of training data, process 200 can store the data in the target sensor training data storage device 102. In some embodiments, the second set of training data includes data items that capture or record the same stimuli as the data items captured in the first set of training data.

[0030] Process 200 trains the generative adversarial model 120 to modify sensor data from the first environmental sensor to be similar to sensor data from the target environmental sensor (at 206). Specifically, process 200 trains the generative adversarial network to modify the input sensor data from the first environmental sensor by reducing the difference in sensor data quality (e.g., resolution) between the input sensor data and the sensor data generated by the target environmental sensor. Process 200 uses the sub-operations 208-216 described below to train the generative adversarial network. For illustration, resolution is used as the sensor data quality to describe operations 208-216. It should be understood that these operations can also be performed using any other quality of the sensor data.

[0031] Process 200 obtains a modified set of sensor data (at 208) from the generator model 106 of the generative adversarial network 120. In some embodiments, process 200 inputs one or more data items from the first sensor data storage device 104 into the generator model 106. In some embodiments, before inputting the data item from the first sensor data storage device 104 into the generator model 106, process 200 inputs the data item into an upscaler that upsamples the sensor data of the data item. Then, process 200 inputs the upsampled data item into the generator model. Using this input data and the corresponding data item in the target sensor data 102 (capturing the same stimulus), the generator model 106 generates a modified set of sensor data 110 having a resolution different from the first resolution and similar to the resolution of the data item from the target sensor data 102.

[0032] Process 200 inputs the modified set of sensor data 110 and one or more data items from the target sensor data 102 into the discriminator model 108 (at 210).

[0033] Process 200 classifies the input data (i.e., the data input into the discriminator model in operation 210) as fake or real using the discriminator model (at 212). In some embodiments, using the data item input into the discriminator model 108 (at operation 210), the discriminator model 108 determines whether each input data item has the target resolution (i.e., real data). If the discriminator model 108 determines that the input data item has the target resolution, it classifies the input data item as real. On the other hand, if the discriminator model 108 determines that the input data item does not have the target resolution, it classifies the input data item as fake.

[0034] Process 200 determines whether the classification of the discriminator model (at operation 212) is correct (at 214). In some embodiments, the classification error engine 112 determines whether the classification of the discriminator model 108 is correct. Each data item input into the discriminator model 108 may have a label identifying the source of the input data, e.g., an identifier specifying whether the data item is provided by the target sensor data storage device 102 or by the generator model 106. Using the label, the classification engine 112 identifies the source of the input data item.

[0035] If the discriminator model 106 classifies the data retrieved from the target sensor data storage device 108 as real, the classification error engine 112 determines that the discriminator 108 is correct. And, if the discriminator model 108 classifies the data retrieved from the generator model 106 as fake, the classification error engine 112 determines that the discriminator 106 is correct. In some embodiments, the classification error engine 112 can use the classification to generate a classification error score ranging from 0 to 1, where 0 indicates that the classification of the discriminator model 108 is incorrect and 1 indicates that the classification of the discriminator model 108 is correct.

[0036] On the other hand, if the discriminator model 108 classifies the data received from the generator model 106 as real, the classification error engine 112 determines that the discriminator 106 is incorrect. In some embodiments, the classification error engine 112 can generate a classification score ranging from 0 to 1, where 0 indicates that the classification of the discriminator model 108 is incorrect and 1 indicates that the classification of the discriminator model 108 is correct.

[0037] Process 200 adjusts the generator model and the discriminator model (at 216). In both cases - that is, when the classification error engine 112 determines that the discriminator model 108 correctly classifies the input data from the generator model 106 as fake and when the classification error engine 112 determines that the discriminator model 108 incorrectly classifies the input data from the generator as real - process 200 adjusts the discriminator model and the generator model. In some embodiments, the classification error score generated by the classification error engine 112 is provided to the discriminator model 108 and the generator model 106. Both models use the classification error to adjust their respective models. The generator model 106 is adjusted based on the classification error so as to subsequently generate modified sensor data from the first sensor data 104, which further reduces the difference in resolution between the sensor data generated by the first environmental sensor and the sensor data generated by the target environmental sensor. The discriminator model 108 is adjusted based on the classification error so as to subsequently be able to distinguish the modified sensor data generated by the generator model 106 from the target sensor data generated by the target environmental sensor.

[0038] To train the generative adversarial model 120, operations 208 - 216 can be iteratively performed a threshold number of times. Alternatively, the training of the generative adversarial model 120 can continue (i.e., this results in the iterative execution of operations 208 - 216) until the discriminator model 108 identifies the modified sensor data as real data. Alternatively, the training of the generative adversarial model 120 can continue until both the discriminator model 108 and the generator model 106 have processed all the data in the first sensor training data storage device 104 and the target sensor training data storage device 102.

[0039] Figure 3 FIG. 300 is a block diagram of an example environment in which a generative adversarial model 120 is used to modify sensor data received from a first environmental sensor.

[0040] As Figure 3 shown, the generative adversarial model 120 (such as trained using the operations and components described with reference to Figure 1 and Figure 2 ) is used to modify first sensor data 302 generated by a first environmental sensor to generate modified sensor data 306. Figure 3 The trained generative adversarial model 120 can be implemented, for example, in a device that includes a first environmental sensor (such as a camera). In such an implementation, the first sensor data 302 is generated by the first environmental sensor of the device, and the device uses the generative adversarial model 120 stored on the device to generate modified sensor data 306. Alternatively, the trained generative adversarial model 120 can be implemented in a computer system separate from the device, where the device includes a first environmental sensor that generates the first sensor data 302. In such an implementation, the first sensor data 302 can be provided by the device, for example, over a network (such as a LAN, WAN, the Internet, or a combination thereof), to the computer system that stores the generative adversarial model 120. The computer system can then use the generative adversarial model 120 to generate modified sensor data 306, and then provide the modified sensor data 306 to the device. The modified sensor data 306 can be used in one or more applications on the device to provide one or more services to a user. For example, in the case where the environmental sensor is an image, video, and / or audio sensor, the modified image, video, and / or audio media can be output from the device, for example, via a screen and / or speaker connected to the device. In other examples, the modified sensor data 306 can be used to determine an estimate of the location and / or orientation of the device.

[0041] With reference to Figure 4 FIGS. Figure 3 the operations of the components for generating the modified sensor data 306 are shown and described.

[0042] Figure 4 FIG. 400 is a flow diagram of an example process 400 for using a generative adversarial model to modify sensor data generated by a first environmental sensor. For illustration, process 400 is described as being executed on a user device (such as a computer, mobile device, camera) that stores a trained generative adversarial model 110. The operations of process 400 can be performed by any suitable device or system (such as any suitable data processing means). The operations of process 400 can also be implemented as instructions stored on a non-transitory computer-readable medium. Execution of the instructions causes one or more data processing means to perform the operations of process 400.

[0043] Process 400 receives first sensor data generated by a first environmental sensor having a first resolution (at 402). In some embodiments, a first environmental sensor (e.g., an image sensor) of a user device (e.g., a camera, a mobile phone) generates first sensor data 302 (e.g., image sensor data) of a particular resolution (e.g., the same resolution as the maximum resolution of the image sensor) in response to an environmental stimulus. The first environmental sensor of the device sends the first image sensor data 302 to the generative adversarial model 120 stored on the user device. As a result, the generative adversarial model 120 receives the first sensor data 302.

[0044] Process 400 determines known defects associated with the first environmental sensor (at 404). In some embodiments, the generative adversarial model 120 may include a known defect storage device 304. The known defect storage device 304 may store known defects of different environmental sensors (e.g., stuck pixels, chromatic aberration). In some embodiments, the known defect storage device 304 may store a known defect data structure that identifies different environmental sensors (e.g., sensor identifiers for the sensors in use), the corresponding defects known for the sensors, and the execution steps for fixing the defect. In other embodiments, the known defect storage device 304 may store a known defect data structure that identifies different environmental sensors (e.g., sensor identifiers for the sensors in use) and the corresponding defects known for the sensors. The known defect engine 308 uses the sensor identifier of the first environmental sensor to identify the defect (if any) of the first environmental sensor from the known defect data structure.

[0045] If the known defect engine 308 identifies a defect of the first environmental sensor from the known defect data structure, the known defect engine 308 adjusts the first sensor data 302 to correct the identified defect by identifying and executing the corresponding steps for correcting the defect (as stored in the known defect data structure) (at 406). In some embodiments, the known defect engine 308 obtains the necessary steps for fixing the defect from the known defect data structure and then executes these steps to correct the known defect. In some embodiments, the known defect engine 308 uses the identified defect to identify a particular generative adversarial model that has been trained to correct the defect. In such embodiments, the particular generative adversarial model is trained using a set of sensor data from the first sensor having the identified defect and a set of sensor data from the first sensor not having the defect. Using the above reference Figure 2A process that is the same as the operation shown and described can use these data sets to train such a generative adversarial model to reduce defects in sensor data. Thus, after inputting the defective first sensor data into such a generative adversarial model, the model generates adjusted first sensor data with reduced or corrected defects in the sensor data. It should be understood that different generative adversarial models can be trained and implemented for each defect. It should also be understood that a single generative adversarial model can be trained and implemented to correct all known defects in the input sensor data.

[0046] After the known defect engine 308 generates the adjusted first sensor data, the known defect engine 308 inputs the adjusted first sensor data into the generator model 106 of the generative adversarial model 120 (at 408).

[0047] However, if the known defect engine 308 does not identify any defects in the first environmental sensor in the known defect data structure, the known defect engine 308 inputs the first sensor data into the generator model 106 of the generative adversarial model 120 (at 410).

[0048] Based on the input first sensor data 302 or the adjusted first sensor data, the generator model 106 generates modified sensor data 306 (at 412). Since the generator model 106 has been trained using the operations described above with reference to Figure 1 and Figure 2 the resolution of the modified sensor data 306 is different from the first resolution and is closer (if not substantially the same) to the target resolution.

[0049] Figure 5 is a block diagram of an example computer system 500 that can be used to perform the above operations. The system 500 includes a processor 510, a memory 520, a storage device 530, and an input / output device 540. Each of the components 510, 520, 530, and 540 can be interconnected, for example, using a system bus 550. The processor 510 is capable of processing instructions executed within the system 500. In some embodiments, the processor 510 is a single-threaded processor. In another embodiment, the processor 510 is a multi-threaded processor. The processor 510 is capable of processing instructions stored in the memory 520 or the storage device 530.

[0050] The memory 520 stores information within the system 500. In one embodiment, the memory 520 is a computer-readable medium. In some embodiments, the memory 520 is a volatile memory unit. In another embodiment, the memory 520 is a non-volatile memory unit.

[0051] The storage device 530 can provide large-capacity storage for the system 500. In some embodiments, the storage device 530 is a computer-readable medium. In various different embodiments, the storage device 530 can include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices (e.g., a cloud storage device) via a network, or some other large-capacity storage device.

[0052] The input / output device 540 provides input / output operations for the system 500. In some embodiments, the input / output device 540 can include one or more of the following: a network interface device (e.g., an Ethernet card), a serial communication device (e.g., one associated with an RS-232 port), and / or a wireless interface device (e.g., one associated with an 802.11 card). In another embodiment, the input / output device can include a drive device configured to receive input data and send output data to other input / output devices (e.g., a keyboard, a printer, and the display device 560). However, other embodiments can also be used, such as mobile computing devices, mobile communication devices, set-top box TV client devices, etc.

[0053] Although example processing systems have been described in Figure 5 , embodiments of the subject matter and functional operations described in this specification can be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them.

[0054] Embodiments of the subject matter and operations described in this specification can be implemented in digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium (or medium) for execution by, or to control the operation of, a data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated for encoding information to be transmitted to a suitable receiver device for execution by the data processing apparatus. The computer storage medium can be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Further, when the computer storage medium is not a propagated signal, the computer storage medium can be the source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0055] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0056] The term "data processing apparatus" encompasses all kinds of devices, equipment, and machines for processing data, including, by way of example, programmable processors, computers, system-on-chips, or multiples or combinations of the foregoing. The apparatus may include dedicated logic circuitry, such as, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program being discussed, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and the execution environment may implement various different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0057] A computer program (which may also be referred to as a program, software, a software application, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, a component, a subroutine, an object, or other units suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. The program may be stored in a part of a file that holds other programs or data (such as, for example, one or more scripts stored in a markup language document), in a single file dedicated to the program being discussed, or in multiple cooperating files (such as, for example, files that store one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0058] The processes and logical flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logical flows can also be performed by dedicated logic circuitry, and the apparatus can also be implemented as dedicated logic circuitry, such as, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0059] For example, processors suitable for executing computer programs include both general and special purpose microprocessors. Typically, a processor will receive instructions and data from a read only memory or a random access memory or both. The basic elements of a computer are a processor for performing actions in accordance with the instructions and one or more memory devices for storing the instructions and data. Typically, a computer will also include one or more mass storage devices (such as, magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operatively coupled to receive data therefrom, or to send data thereto, or both. However, a computer need not have such devices. In addition, a computer may be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (such as, a universal serial bus (USB) flash drive), etc. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices (such as, EPROM, EEPROM, and flash memory devices); magnetic disks (such as, internal hard disks or removable disks); magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0060] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (such as, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (such as, a mouse or a trackball) by which the user may provide input to the computer. Other kinds of devices may also be used to provide for interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as, visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including, sound, voice, or tactile input. In addition, a computer may interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a web page to a web browser on a client device of the user in response to a request received from the web browser.

[0061] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a backend component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a frontend component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), intranets (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0062] A computing system can include a client and a server. The client and the server are typically located remotely from each other and typically interact through a communication network. The relationship between the client and the server is created by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a client device (e.g., for the purpose of displaying the data to a user interacting with the client device and receiving user input from the user interacting with the client device). Data generated at the client device (e.g., the result of a user interaction) can be received at the server from the client device.

[0063] Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what is claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features may be described as acting in certain combinations and even initially claimed as such, in some cases, one or more features from a claimed combination can be deleted from the combination, and the claimed combination can be directed to a sub-combination or a variant of a sub-combination.

[0064] Similarly, although operations are depicted in the drawings in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the foregoing embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged into multiple software products.

[0065] Accordingly, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the acts recited in the claims can be performed in a different order and still achieve the desired results. Additionally, the processes described in the figures need not be in the particular order or sequential order shown to achieve the desired results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method, comprising: Receiving a first set of training data generated by a first environmental sensor having a first quality, the first set of training data having the first quality; Receiving a second set of training data generated by a target environmental sensor having a target quality higher than the first quality, the second set of training data having the target quality, wherein the target environmental sensor generates data of the same type as the first environmental sensor; Using the first set of training data and the second set of training data to train a generative adversarial model to modify sensor data from the first environmental sensor, wherein the training comprises: Obtaining, from a generator model of the generative adversarial model and using one or more data items from the first set of training data, a modified set of sensor data having a second quality different from the first quality, wherein the modified set of sensor data is of the same type as the first set of training data; Inputting a set of data items including one or more data items from the second set of training data and the modified set of sensor data into a discriminator model of the generative adversarial model; Determining, by the discriminator model and using the set of data items, a classification of each data item, the classification indicating whether the data item is derived from the modified set of sensor data or the second set of training data; Determining a classification error based on the classification of each data item; Adjusting the discriminator model and the generator model based on the classification error, wherein the first quality and the second quality include one or more of error rate, fidelity, and / or signal-to-noise ratio.

2. The computer-implemented method according to claim 1, wherein, Each of the first environmental sensor and the target environmental sensor acquires one of sound, image, or video.

3. The computer-implemented method according to claim 1, further comprising: Receiving first sensor data generated by the first environmental sensor having the first quality; Inputting the first sensor data into the generative adversarial model; and Using the generator model of the generative adversarial model to obtain modified sensor data based on the input first sensor data.

4. The computer-implemented method according to claim 1, further comprising: Receiving first sensor data generated by the first environmental sensor having the first quality; Inputting information about the first environmental sensor into a known defect data structure storing known defects of different environmental sensors; Obtaining the known defects of the first environmental sensor from the known defect data structure; Adjusting the first sensor data based on the known defects of the first environmental sensor; Inputting the adjusted first sensor data into the generative adversarial model; and Using the generator model of the generative adversarial model to obtain modified sensor data based on the adjusted first sensor data.

5. The method according to any one of the preceding claims, wherein, The first environmental sensor corresponds to a defective version of the target environmental sensor.

6. A system for modifying sensor data, comprising: One or more memory devices that store instructions; and One or more data processing devices, configured to interact with the one or more memory devices and, when executing instructions, perform operations including: Receiving a first set of training data generated by a first environmental sensor having a first quality, the first set of training data having the first quality; Receiving a second set of training data generated by a target environmental sensor having a target quality that is higher than the first quality, the second set of training data having the target quality, wherein the target environmental sensor generates data of the same type as the first environmental sensor; Using the first set of training data and the second set of training data to train a generative adversarial model to modify sensor data from the first environmental sensor, wherein the training includes: Obtaining, from a generator model of the generative adversarial model and using one or more data items from the first set of training data, a modified set of sensor data having a second quality different from the first quality, wherein the modified set of sensor data is of the same type as the first set of training data; Inputting a set of data items including the second set of training data and one or more data items from the modified set of sensor data into a discriminator model of the generative adversarial model; Determining, by the discriminator model and using the set of data items, a classification for each data item, the classification indicating whether the data item is derived from the modified set of sensor data or the second set of training data; Determining a classification error based on the classification of each data item; Adjusting the discriminator model and the generator model based on the classification error, wherein the first quality and the second quality include one or more of error rate, fidelity, and / or signal-to-noise ratio.

7. The system according to claim 6, wherein Each of the first environmental sensor and the target environmental sensor acquires one of sound, image, or video.

8. The system according to claim 6, wherein The operations that the one or more data processing devices are configured to perform further include: Receiving first sensor data generated by the first environmental sensor having the first quality; Inputting the first sensor data into the generative adversarial model; and Using the generator model of the generative adversarial model to obtain modified sensor data based on the input first sensor data.

9. The system according to claim 6, wherein, The operations that the one or more data processing devices are configured to perform further include: Receiving first sensor data generated by the first environmental sensor having the first quality; Inputting information about the first environmental sensor into a known defect data structure that stores known defects of different environmental sensors; Obtaining the known defects of the first environmental sensor from the known defect data structure; Adjusting the first sensor data based on the known defects of the first environmental sensor; Inputting the adjusted first sensor data into the generative adversarial model; and Using the generator model of the generative adversarial model to obtain modified sensor data based on the adjusted first sensor data.

10. The system according to any one of claims 6-9, wherein, The first environmental sensor corresponds to a defective version of the target environmental sensor.

11. A non - transitory computer - readable medium storing instructions that, when executed by one or more data - processing devices, cause the one or more data - processing devices to perform operations including: Receiving a first set of training data generated by a first environmental sensor having a first quality, the first set of training data having the first quality; Receiving a second set of training data generated by a target environmental sensor having a target quality higher than the first quality, the second set of training data having the target quality, wherein the target environmental sensor generates data of the same type as the first environmental sensor; Using the first set of training data and the second set of training data to train a generative adversarial model to modify sensor data from the first environmental sensor, wherein the training includes: Obtaining, from a generator model of the generative adversarial model and using one or more data items from the first set of training data, a modified set of sensor data having a second quality different from the first quality, wherein the modified set of sensor data is of the same type as the first set of training data; Inputting a set of data items including one or more data items from the second set of training data and the modified set of sensor data into a discriminator model of the generative adversarial model; Determining, by the discriminator model and using the set of data items, a classification for each data item, the classification indicating whether the data item is from the modified set of sensor data or the second set of training data; Determining a classification error based on the classification of each data item; Adjusting the discriminator model and the generator model based on the classification error, wherein the first quality and the second quality include one or more of error rate, fidelity, and / or signal - to - noise ratio.

12. The non-transitory computer-readable medium according to claim 11, wherein, Each of the first environmental sensor and the target environmental sensor acquires one of sound, image, or video.

13. The non-transitory computer-readable medium according to claim 11, wherein, The instructions cause the one or more data - processing devices to perform operations including those of the computer - implemented method of claim 1, and the operations further include: Receiving first sensor data generated by the first environmental sensor having the first quality; Inputting the first sensor data into the generative adversarial model; and Using the generator model of the generative adversarial model to obtain modified sensor data based on the input first sensor data.

14. The non-transitory computer-readable medium according to claim 11, wherein, The instructions cause the one or more data - processing devices to perform operations including those of the computer - implemented method of claim 1, and the operations further include: Receiving first sensor data generated by the first environmental sensor having the first quality; Inputting information about the first environmental sensor into a known - defect data structure storing known defects of different environmental sensors; Obtaining the known defects of the first environmental sensor from the known - defect data structure; Adjusting the first sensor data based on the known defects of the first environmental sensor; Inputting the adjusted first sensor data into the generative adversarial model; and Using the generator model of the generative adversarial model, obtain modified sensor data based on the adjusted first sensor data.

15. The non-transitory computer-readable medium according to any one of claims 11-14, wherein, The first environmental sensor corresponds to a defective version of the target environmental sensor.

Images