Optical device

Abstract

An optical device comprising: an optical sensor; one or more diffusive optical elements disposed on a surface of a display screen and configured to diffuse and distribute a first portion of light associated with an object across an array of sensor elements of the optical sensor; and one or more processors. The display screen comprises one or more transmissive components that allow the first portion of light to pass through the display screen via the surface of the display screen, and one or more blocking components that prevent a second portion of light associated with the object from passing through the display screen. The one or more processors are configured to obtain, from the optical sensor, sensor data associated with the first portion of light, process the sensor data to determine an image of the object, and perform one or more actions based on the image of the object.

Description

Background Technology

[0001] Optical devices can be used to capture information about light. For example, an optical device can capture information related to a set of wavelengths associated with light. An optical device may include a set of sensor elements (e.g., optical sensors, spectral sensors, and / or image sensors) for capturing information. For example, an array of sensor elements can be used to capture information related to multiple wavelengths. Summary of the Invention

[0002] In some embodiments, an optical device includes: an optical sensor including an array of sensor elements; one or more diffuse optical elements disposed on a surface of a display screen and configured to diffuse and distribute a first portion of light associated with an object across the array of sensor elements of the optical sensor, wherein the display screen includes one or more transmissive members that allow the first portion of light to pass through the display screen via a surface of the display screen, and wherein the display screen includes one or more blocking members that prevent a second portion of light associated with the object from passing through the display screen; and one or more processors configured to: acquire sensor data associated with the first portion of light from the optical sensor; process the sensor data to determine an image of the object; and perform one or more actions based on the image of the object.

[0003] In some embodiments, a non-transitory computer-readable medium storing instructions, the instructions including one or more instructions, when executed by one or more processors of an optical device, causing the one or more processors to: obtain sensor data from an optical sensor of the optical device associated with a portion of light, the portion of light being associated with an object and diffused and distributed across the input surface of the optical sensor by one or more diffusing optical elements disposed on the surface of a display screen, wherein the display screen includes one or more transmissive elements that allow the portion of light to pass through the display screen, and wherein the display screen includes one or more blocking elements that prevent different portions of the light associated with the object from passing through the display screen; process the sensor data to determine an image of the object; and provide the image of the object.

[0004] In some embodiments, a method includes: obtaining sensor data associated with a portion of light, which is associated with an object, via and from an optical sensor of an optical device, transmitted via a display screen, and diffused and distributed in a caustic pattern across the input surface of the optical sensor by one or more diffusing optical elements; processing the sensor data by the optical device and based on the caustic pattern to determine an image of the object; and performing one or more actions by the optical device based on the image of the object. Attached Figure Description

[0005] Figures 1A to 1B This is a diagram of an example implementation described herein.

[0006] Figure 2 This is a diagram illustrating an example of using images of defined objects to train and use a machine learning model.

[0007] Figure 3 This is a diagram of an example environment in which the systems and / or methods described in this paper can be implemented.

[0008] Figure 4 yes Figure 3 A diagram of example components of one or more devices.

[0009] Figures 5 to 7 This is a flowchart of an example process related to optical equipment. Detailed Implementation

[0010] The following detailed description of exemplary embodiments refers to the accompanying drawings. The same reference numerals in different drawings may identify the same or similar elements.

[0011] User equipment (such as mobile phones) includes a display screen (e.g., on the front surface of the user device) for displaying information to the user. Typically, user devices include a conventional front-facing camera located next to the display screen on the front surface of the user device. For example, a conventional front-facing camera may be located on the front surface between the display screen and the bezel. A conventional front-facing camera can be used to acquire images of objects, such as for identification and / or authentication applications (e.g., to determine whether a user of the user device is allowed to interact with the user device). However, due to the footprint of a conventional front-facing camera, placing it next to the display screen increases the user device's footprint and results in the user device including a "blank" area without a display screen on the front surface of its display.

[0012] Furthermore, recent attempts to place a traditional front-facing camera behind the display have been unsuccessful because the display's circuitry obstructs part of the camera and reduces the amount of light reaching it. This results in incomplete, inaccurate, and / or low-quality images of the object captured by the front-facing camera. Such images are unsuitable for identification and / or authentication applications. Additionally, traditional front-facing cameras include an imaging lens, requiring a certain thickness to prevent them from being positioned behind the display, which is necessary for user devices requiring a small footprint.

[0013] Some embodiments described herein provide an optical device comprising: an optical sensor; one or more diffuse optical elements disposed on the surface of a display screen (e.g., a user device) and configured to diffuse and distribute a first portion of light associated with an object across an array of sensor elements of the optical sensor; and one or more processors. The display screen may include one or more transmissive members that allow the first portion of light to pass through the display screen via a surface of the display screen; and one or more blocking members that prevent a second portion of the light associated with the object from passing through the display screen. One or more processors may be configured to acquire sensor data associated with the first portion of light (e.g., a portion of light that has already passed through the display screen) from the optical sensor. One or more processors may process the sensor data to determine an image of the object. For example, one or more processors may use an algorithm to process the sensor data, the algorithm being configured to construct an image of the object based on a pattern such as a caustic pattern, wherein one or more diffuse optical elements diffuse and distribute the first portion of light across an array of sensor elements of the optical sensor to determine the image of the object. One or more processors may perform one or more actions based on the image of the object (e.g., analyzing the image to identify and / or verify the object).

[0014] Thus, the optical device described herein can determine a complete, accurate, and high-quality image of an object based solely on the first portion of the light passing through the display (e.g., for identification and / or authentication applications), which is impossible with conventional front-facing cameras. Furthermore, no modifications to the display's circuitry are required (e.g., reducing the density of pixel components on the optical device) to allow the optical device to determine the image of the object. This allows for a consistent density of pixel components on the display, thereby reducing the design complexity associated with designing, assembling, and / or maintaining the display, the optical device, and / or the user device. Additionally, the optical device does not include a lens, making it thinner than conventional front-facing cameras. This allows the optical device to be incorporated into user devices requiring a small form factor (e.g., located behind the display), which might be impossible with conventional front-facing cameras.

[0015] Figures 1A to 1B This is an overview diagram of the example implementation 100 described herein. (As shown...) Figure 1A As shown, example embodiment 100 includes a display screen 102 and an optical sensor 104. The display screen 102 may be a liquid crystal display (LCD), a light-emitting diode (LED) display, or an organic light-emitting diode (OLED) display, etc. The display screen 102 and / or the optical sensor 104 may be associated with optical devices, which are described in more detail elsewhere herein.

[0016] like Figure 1AAs further shown, the display screen 102 may include one or more pixel components 106. The pixel components 106 may be configured to emit light associated with a specific wavelength or wavelength range (e.g., via the display surface of the display screen 102). For example, as Figure 1A As shown, a first pixel component 106 with a gray shading can be configured to emit light associated with a first wavelength range (e.g., red light), a second pixel component 106 with a diamond pattern can be configured to emit light associated with a second wavelength range (e.g., blue light), and a third pixel component 106 with a diagonal pattern can be configured to emit light associated with a third wavelength range (e.g., green light). In some embodiments, pixel components 106 can be configured to allow light (e.g., ambient light) to pass through pixel components 106 (e.g., when pixel components 106 are emitting or not emitting light). For example, ambient light can be transmitted through pixel components 106 through the display surface of display screen 102 (e.g., reaching optical sensor 104).

[0017] One or more pixel components 106 may be arranged in a pixel component configuration on the display screen 102. For example, Figure 1A Pixel component configurations 108 and 110 are shown, but other pixel component configurations are also considered. For example... Figure 1A As shown, the pixel configuration 108 can group a set of pixel components 106 (e.g., including a first pixel component 106, a second pixel component 106, and a third pixel component 106) in a region on the display 102. One or more gaps 112 may exist between a set of pixel components 106 in this region.

[0018] In some implementations, the gap 112 may be transparent or otherwise configured to allow light (e.g., ambient light) to pass through the gap 112. For example, ambient light may be transmitted through the gap 112 through the display surface of the display screen 102 (e.g., to the optical sensor 104).

[0019] As further shown in pixel component configuration 108, display screen 102 may include one or more circuit components 114. Circuit components 114 may include circuitry or other structures and / or elements configured to facilitate operation of display screen 102 and / or one or more pixel components 106. In some embodiments, circuit components 114 may block or otherwise be configured to prevent light (e.g., ambient light) from passing through circuit components 114. For example, circuit components 114 may prevent ambient light from being transmitted through circuit components 114 through the display surface of display screen 102 (e.g., reaching optical sensor 104).

[0020] like Figure 1AAs further shown, the pixel component configuration 110 allows a group of pixel components 106 (e.g., including a first pixel component 106, a second pixel component 106, and a third pixel component 106) to form a group with one or more window components 116 in a region on the display screen 102. The window component 116 may be transparent or otherwise configured to allow light (e.g., ambient light) to pass through the window component 116. For example, ambient light may be transmitted through the window component 116 through the surface of the display screen 102 (e.g., to the optical sensor 104). In some embodiments, the window component 116 may be configured to allow light associated with one or more wavelengths or wavelength ranges to pass through the window component 116 and / or prevent light associated with one or more different wavelengths or different wavelength ranges from passing through the window component 116. In some embodiments, the window component 116 may include one or more diffuse optical elements (e.g., included in or separate from one or more diffuse optical elements 122 described herein) to diffuse light passing through the window component 116.

[0021] like Figure 1B As shown, the display screen 102 may include one or more transmissive elements 118 and / or one or more blocking elements 120. The one or more transmissive elements 118 may include one or more pixel elements 106, one or more gaps 112, one or more window elements 116, and / or other components of the display screen 102 that allow light to pass through the display screen 102 (e.g., via the display surface of the display screen 102). The one or more blocking elements 120 may include one or more circuitry elements and / or other components of the display screen 102 that prevent light from passing through the display screen 102 (e.g., via the display surface of the display screen 102).

[0022] In some embodiments, one or more diffuse optical elements 122 may be disposed between the display screen 102 and the optical sensor 104 (e.g., the input surface of the optical sensor 104). The diffuse optical element 122 may include a light diffuser and / or any other type of optical structure for diffusing light. For example, one or more diffuse optical elements 122 may be configured to diffuse and distribute light passing through the display screen 102 via one or more transmission elements 118 across the input surface of the optical sensor 104. In some embodiments, one or more diffuse optical elements 122 may be configured to diffuse and distribute light across the input surface of the optical sensor 104 in a pattern (e.g., a caustic pattern associated with at least one point spread function (PSF)). Figure 1BAs shown, one or more diffuse optical elements 122 may be disposed on the surface of the display screen 102, such as the "back" surface of the display screen 102 (e.g., a non-display surface). In some embodiments, one or more diffuse optical elements 122 may be disposed on the display surface of the display screen 102 or may be integrated into the display screen 102 (e.g., on one or more pixel components 106, one or more gaps 112, one or more window components and / or the like within the housing of the display screen 102).

[0023] like Figure 1B As further shown, the optical sensor 104 may include one or more sensor elements 124 (e.g., an array of sensor elements, also referred to herein as a sensor array), each sensor element being configured to acquire information. For example, sensor element 124 may provide an indication of the intensity of light incident on sensor element 124 (e.g., active / inactive or more fine-grained intensity indication). As another example, sensor element 124 may provide an indication of the wavelength or wavelength range of light incident on sensor element 124 (e.g., red, blue, green, ultraviolet, infrared, and / or the like). The optical sensor 104 may be configured to collect information acquired by one or more sensor elements 124 to generate sensor data.

[0024] In one example, optical devices (e.g., including display 102, one or more diffuse optical elements 122, and / or optical sensors 104) can be configured to capture objects 126 (e.g., potted plants, such as...). Figure 1B Information related to (as shown). For example... Figure 1B As shown, light 128 may originate from one or more points of object 126 (e.g., light reflected from one or more points of object 126 from a light source) and may be received by an optical device (e.g., object 126 may be in the field of view of the optical device). A first portion of light 128 may enter display screen 102 via the display surface of display screen 102 and may pass through display screen 102 by means of one or more transmission members 118 (e.g., may be allowed to transmit through display screen 102 to the rear surface of display screen). A second portion of light 128 may be blocked by one or more blocking members 120 and may not pass through display screen 102.

[0025] The first portion 128 of light can be diffused by one or more diffuse optical elements 122 (e.g., before the first portion of light passes through display screen 102, as the first portion of light passes through display screen 102, and / or after the first portion of light passes through display screen 102, etc.) to form diffuse light 130. The diffuse light 130 can be distributed across one or more sensor elements 124 of optical sensor 104 by one or more diffuse optical elements 122. In some embodiments, the diffuse light 130 can be patterned (e.g., Figure 1B The diffuse light pattern shown is distributed across one or more sensor elements 124, such as a caustic pattern.

[0026] like Figure 1B As further shown, the optical device can be associated with one or more processors 132 and can provide sensor data to one or more processors 132 as indicated by reference numeral 134. The sensor data can indicate information related to light 128 originating from object 126 and / or diffuse light 130 distributed across one or more sensor elements 124. For example, one or more sensor elements 124 can generate sensor data based on receiving diffuse light 130. The optical sensor 104 can transmit the sensor data to one or more processors 132.

[0027] like Figure 1B As further shown by reference numeral 136, one or more processors 132 can process sensor data to determine (e.g., generate, construct, and / or create, etc.) an image of object 126 (e.g., an image of a potted plant). For example, one or more processors 132 can identify (e.g., by searching data structures associated with one or more processors 132) an algorithm for constructing an image based on light diffused by one or more diffuse optical elements 122, and can use the algorithm to process sensor data to determine an image of object 126. In some embodiments, the algorithm can be configured to construct an image of object 126 based on a pattern (e.g., a caustic pattern), wherein one or more diffuse optical elements 122 distribute diffuse light 130 onto one or more sensor elements 124 in this pattern.

[0028] In some implementations, one or more processors 132 may determine calibration information associated with the circuitry of the display 102. The calibration information may indicate positional information associated with one or more transmissive elements 118, and / or one or more blocking elements 120, and / or may indicate how one or more diffuse optical elements 122 distribute diffuse light across one or more sensor elements 124 based on the positional information. Therefore, one or more processors 132 may optimize algorithms based on the calibration information. For example, one or more processors 132 may identify, based on the calibration information, a set of sensor elements 124 associated with a subset of sensor data that is more likely to contribute to accurately constructing an image than other sensor elements 124. Therefore, one or more processors 132 may optimize algorithms to support this set of sensor elements 124 rather than other sensor elements 124 to determine an image of object 126 based on the sensor data. In this way, one or more processors 132 can generate images of object 126 that are more accurate than those using unoptimized algorithms and / or can save computational resources (e.g., processing resources, memory resources, communication resources and / or power resources, etc.) that would otherwise be consumed when using unoptimized algorithms.

[0029] In some implementations, the sensor data may include a first set of data associated with diffuse light 130 and a second set of data associated with additional light, such as light 102 emitted by one or more pixel elements 106 of the display screen. One or more processors 132 may process the sensor data to identify the first and second sets of sensor data, and may process the first set of sensor data (e.g., using the algorithm described above) without processing the second set of sensor data to determine an image of object 126. For example, one or more processors 132 may computationally remove (e.g., associated with light emitted by one or more pixel elements 106) the second set of sensor data before processing the sensor data to determine an image of object 126. In this way, one or more processors 132 may generate an image of object 126 based on light 128 originating from object 126, thereby providing a more accurate image of object 126 than an image of object 126 otherwise determined based on all light received by one or more sensor elements 124.

[0030] In some implementations, one or more processors 132 may use a machine learning model to process sensor data to determine an image of object 126. The machine learning model may have been trained on training data that may include data from example sensors, information relating to example patterns of diffuse light (e.g., caustic patterns) across the distribution of sensor elements associated with the example sensor data, and / or example images associated with the example sensor data. Using the training data as input to the machine learning model, the model can be trained to recognize one or more relationships (e.g., between example sensor data, information relating to example patterns, and / or example images) to determine the image of the object. The machine learning model may be configured to process sensor data in a manner similar to the following... Figure 2 The described method is used to train and / or use the machine learning model. One or more processors 132 may include one or more devices (e.g., one or more data storage devices) that store the machine learning model.

[0031] In some implementations, one or more processors 132 may cause an image of object 126 to be displayed on display screen 102. For example, one or more processors 132 may send an image of object 126 to display screen 102 to cause display screen 102 to display an image of object 126. As another example, when display screen 102 is included in another device such as a user device, one or more processors 132 may send an image of object 126 to the user device to cause the user device to display an image of object 126 on display screen 102.

[0032] In some implementations, one or more processors 132 may use biometric analysis to process images of object 126. For example, one or more processors 132 may use facial recognition analysis to process images of object 126 (e.g., when object 126 is the face of an optical device user). Additionally or alternatively, one or more processors 132 may use vein recognition analysis and / or hand recognition analysis (e.g., for body parts) to process images of object 126. In some implementations, one or more processors 132 may acquire multiple images of object 126 and may use pose analysis, and / or motion analysis, etc., to process these images. In some implementations, one or more processors 132 may send one or more images of object 126 to another device, such as a user device associated with display 102, to analyze one or more images as described above. In this way, one or more processors 132 may facilitate the identification and / or authentication of the user of the optical device (e.g., to allow the user to interact with display 102 and / or the user device associated with display 102).

[0033] In some implementations, when the machine learning model is used to determine an image of object 126, one or more processors 132 can retrain the machine learning model based on the image of object 126. For example, the content system can use the image of object 126 as additional training data to retrain and / or update the machine learning model. In this way, one or more processors 132 can improve the accuracy of the machine learning model, which can improve the speed and / or efficiency of the machine learning model, thereby saving the computational resources of one or more processors 132.

[0034] As mentioned above, Figure 1A and 1B Provided as one or more examples. Other examples may be related to... Figure 1A and Figure 1B The descriptions are different.

[0035] Figure 2 This is a diagram illustrating example 200 of training and using a machine learning model by combining images of a defined object. A machine learning system can be used to perform the training and use of the machine learning model described herein. The machine learning system may include or be included in computing devices, servers, or cloud computing environments, such as one or more processors 132 and / or user devices described in more detail elsewhere herein.

[0036] As shown by reference numeral 205 in the accompanying drawings, an observation set can be used to train a machine learning model. This observation set can be obtained from training data (e.g., historical data), such as data collected during one or more of the processes described herein. In some embodiments, the machine learning system can receive this observation set (e.g., as input) from optical sensor 104, as described elsewhere herein.

[0037] As indicated by reference numeral 210 in the accompanying drawings, the observation set includes a feature set. The feature set may include a set of variables, and the variables may be referred to as features. A specific observation may include a set of variable values ​​(or feature values) corresponding to the variable set. In some implementations, the machine learning system may determine the variables of the observation set and / or the variable values ​​of a specific observation based on input received from the optical sensor 104. For example, the machine learning system may identify a feature set (e.g., one or more features and / or feature values) by extracting a feature set from structured data, by performing natural language processing from unstructured data, and / or by receiving input from an operator.

[0038] As an example, the feature set used for the observation set may include (e.g., a first feature of sensor element group 1 of one or more sensor elements 124), a second feature of sensor element group 2, a third feature of sensor element group 3, and so on. As shown, for a first observation, the first feature may have a value of sensor data 1A, the second feature may have a value of sensor data 2A, the third feature may have a value of sensor data 3A, and so on. These features and feature values ​​are provided as examples and may differ in other examples.

[0039] As indicated by reference numeral 215 in the attached figure, the observation set can be associated with a target variable. The target variable can represent a variable with numerical values, a variable with numerical values ​​falling within a range of values ​​or having some discrete possible values, a variable that can be selected from one of several options (e.g., one of several categories, several classifications, or several labels), and / or a variable with Boolean values. The target variable can be associated with target variable values, and the target variable values ​​can be observation-specific. In Example 200, the target variable is (e.g., associated with a feature set) an image of an object that has a value for image A for a first observation.

[0040] The target variable can represent the value that a machine learning model is trained to predict, and the feature set can represent the variables that are input into the trained machine learning model to predict the value of the target variable. This observation set can include the values ​​of the target variable, allowing the machine learning model to be trained to identify patterns in the feature set that lead to the values ​​of the target variable. The machine learning model trained to predict the value of the target variable can be called a supervised learning model.

[0041] In some implementations, the machine learning model can be trained on an observation set that does not include the target variable. This can be called an unsupervised learning model. In this case, the machine learning model can learn patterns from the observation set without labels or supervision, and can provide outputs that indicate such patterns, such as identifying relevant itemsets in the observation set by using clustering and / or association.

[0042] As shown by reference numeral 220 in the attached figure, a machine learning system can train a machine learning model using an observation set and one or more machine learning algorithms, such as regression algorithms, decision tree algorithms, neural network algorithms, k-nearest neighbor algorithms, support vector machine algorithms, etc. After training, the machine learning system can store the machine learning model as trained machine learning model 225 for analysis of new observations.

[0043] As shown by reference numeral 230 in the figure, a machine learning system can apply a trained machine learning model 225 to new observations, for example, by receiving new observations and inputting them into the trained machine learning model 225. As illustrated, as an example, a new observation may include a first feature 1X of sensor data, a second feature 2X of sensor data, a third feature 3X of sensor data, and so on. The machine learning system can apply the trained machine learning model 225 to the new observations to generate outputs (e.g., results). The type of output may depend on the type of machine learning model and / or the type of machine learning task being performed. For example, the output may include predicted values ​​of the target variable, as in the case of supervised learning. Additionally or alternatively, the output may include information identifying the cluster to which the new observation belongs and / or information indicating the similarity between the new observation and one or more other observations, as in the case of unsupervised learning.

[0044] As an example, the trained machine learning model 225 can predict the image value X of the target variable of an image of a newly observed object, as indicated by reference numeral 235. Based on this prediction, the machine learning system can provide a first recommendation, provide output for determining the first recommendation, perform a first automatic action, and / or cause the first automatic action to be performed (e.g., by instructing another device to perform the automatic action), etc. The first automatic action may include, for example, displaying an image of the object on display screen 102.

[0045] In some implementations, the trained machine learning model 225 can classify (e.g., cluster) new observations within clusters, as indicated by reference numeral 240. Observations within a cluster may have a threshold similarity. For example, if the machine learning system classifies a new observation into a first cluster (e.g., face images), the machine learning system can provide a first recommendation. Additionally or alternatively, the machine learning system may perform a first automatic action based on the classification of new observations in the first cluster and / or may cause the first automatic action to be performed (e.g., by instructing another device to perform the automatic action), such as the first automatic action described above.

[0046] In some implementations, recommendations and / or automated actions associated with new observations may be based on target variable values ​​with specific labels (e.g., classification or categorization), on whether the target variable values ​​meet one or more thresholds (e.g., whether the target variable values ​​are greater than a threshold, less than a threshold, equal to a threshold, fall within a threshold range, etc.), and / or on clustering that classifies the new observations.

[0047] In this way, machine learning systems can apply a rigorously automated process to determine images of objects based on sensor data. Machine learning systems are able to identify and / or characterize tens, hundreds, thousands, or millions of features and / or tens, hundreds, thousands, or millions of observational feature values, thereby improving accuracy and consistency, and reducing the latency associated with determining images of objects from sensor data compared to the computational resources allocated to manually determining images of objects using features or feature values ​​by tens, hundreds, or thousands of operators.

[0048] As mentioned above, Figure 2 This is provided as an example. Other examples may be provided in conjunction with it. Figure 2 The descriptions are different.

[0049] Figure 3 This is a diagram of example environment 300, in which the systems and / or methods described in this article can be implemented. (See diagram for example.) Figure 3 As shown, environment 300 may include optical device 310, which may include one or more processors 320 (e.g., corresponding to the processors described herein). Figure 1A and 1B One or more processors 126 described herein and an optical sensor 330 (e.g., corresponding to the one or more processors 126 described herein) ... Figure 1A and 1B The optical sensor 104 is described. Environment 300 may also include user equipment 340 and network 350. Devices in environment 300 can be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0050] Optical device 310 may include optical devices capable of identifying, storing, processing, and / or routing images associated with an object. In some embodiments, optical device 310 may be incorporated into user equipment 340. In some embodiments, optical device 310 may receive and / or send information to another device (such as user equipment 340) in environment 300. Optical device 310 may include combined... Figure 4 One or more processors 320 and / or optical sensors 330 are described in more detail.

[0051] Optical sensor 330 includes a device capable of sensing light. For example, optical sensor 330 may include an image sensor, a multispectral sensor, and / or a spectral sensor. In some embodiments, optical sensor 330 may include a silicon (Si)-based sensor, an indium gallium arsenide (InGaAs)-based sensor, a lead sulfide (PbS)-based sensor, or a germanium (Ge)-based sensor, and may utilize one or more sensor technologies, such as complementary metal-oxide-semiconductor (CMOS) technology or charge-coupled device (CCD) technology. In some embodiments, optical sensor 330 may include a front-illuminated (FSI) sensor and / or a back-illuminated (BSI) sensor, etc. In some embodiments, optical sensor 330 may be included in the camera of optical device 310 and / or user equipment 340.

[0052] User equipment 340 includes one or more devices capable of receiving, generating, storing, processing, and / or providing images associated with an object. For example, user equipment 340 may include communication and / or computing devices such as mobile phones (e.g., smartphones, and / or wireless phones), computers (e.g., laptops, tablets, and / or handheld computers), gaming devices, wearable communication devices (e.g., smartwatches, and / or smart glasses), or similar types of devices. In some embodiments, the user equipment may include a display screen (e.g., display screen 102) for displaying images associated with an object. In some embodiments, user equipment 340 may receive and / or send information to another device in environment 300 (e.g., optical device 310).

[0053] Network 350 includes one or more wired and / or wireless networks. For example, network 350 may include cellular networks (e.g., Long Term Evolution (LTE) networks, Code Division Multiple Access (CDMA) networks, 3G networks, 4G networks, 5G networks, and / or another type of next-generation network, etc.), Public Land Mobile Networks (PLMN), Local Area Networks (LAN), Wide Area Networks (WAN), Metropolitan Area Networks (MAN), Telephone Networks (e.g., Public Switched Telephone Network (PSTN)), Private Networks, Self-organizing Networks, Intranets, the Internet, Fiber-based Networks, Cloud Computing Networks, etc., and / or combinations of these or other types of networks.

[0054] Figure 3 The number and arrangement of devices and networks shown are provided as an example. In reality, with... Figure 3 Compared to those shown, there may be additional equipment and / or networks, fewer equipment and / or networks, different equipment and / or networks, or equipment and / or networks with different arrangements. Furthermore, Figure 3The device shown, which comprises two or more devices, can be implemented in a single device, or Figure 3 The single device shown can be implemented as multiple distributed devices. For example, although optical device 310 and user device 340 are described as separate devices, they can be implemented as a single device. Additionally or alternatively, a group of devices (e.g., one or more devices) of environment 300 can perform one or more functions described as being performed by another group of devices of environment 300.

[0055] Figure 4 This is a diagram of example components of device 400, which may correspond to optical device 310 and / or user equipment 340. In some embodiments, optical device 310 and / or user equipment 340 may include one or more devices 400 and / or one or more components of device 400. Figure 4 As shown, device 400 may include bus 410, processor 420, memory 430, storage unit 440, input unit 450, output unit 460, and communication unit 470.

[0056] Bus 410 includes components capable of wired and / or wireless communication between components of device 400. Processor 420 includes a central processing unit, graphics processing unit, microprocessor, controller, microcontroller, digital signal processor, field-programmable gate array, application-specific integrated circuit, and / or another type of processing unit. Processor 420 is implemented in hardware, firmware, or a combination of hardware and software. In some embodiments, processor 420 includes one or more processors that can be programmed to perform functions. Memory 430 includes random access memory, read-only memory, and / or another type of memory (e.g., flash memory, magnetic storage, and / or optical storage).

[0057] Storage component 440 stores information and / or software related to the operation of device 400. For example, storage component 440 may include a hard disk drive, disk drive, optical disk drive, solid-state drive, optical disk, digital multifunction disk, and / or another type of non-transitory computer-readable medium. Input component 450 enables device 400 to receive input, such as user input and / or sensed input. For example, input component 450 may include a touchscreen, keyboard, keypad, mouse, buttons, microphone, switch, sensor, GPS component, accelerometer, gyroscope, and / or actuator. Output component 460 enables device 400 to provide output, such as via a display, speaker, and / or one or more light-emitting diodes. Communication component 470 enables device 400 to communicate with other devices, such as via wired and / or wireless connections. For example, communication component 470 may include a receiver, transmitter, transceiver, modem, network interface card, and / or antenna.

[0058] Device 400 can perform one or more of the processes described herein. For example, a non-transitory computer-readable medium (e.g., memory 430 and / or storage unit 440) can store a set of instructions (e.g., one or more instructions, code, software code, and / or program code) for execution by processor 420. Processor 420 can execute the set of instructions to perform one or more of the processes described herein. In some embodiments, a set of instructions executed by one or more processors 420 causes one or more processors 420 and / or device 400 to perform one or more of the processes described herein. In some embodiments, hardwired circuitry may be used in place of or in combination with instructions to perform one or more of the processes described herein. Therefore, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0059] Figure 4 The number and arrangement of components shown are provided as an example. Device 400 may include components with... Figure 4 The components shown are additional, fewer, different, or differently arranged in comparison. Alternatively, a set of components of device 400 (e.g., one or more components) may perform one or more functions described as being performed by another set of components of device 400.

[0060] Figure 5 This is a flowchart of an example process 500 associated with an optical device (e.g., optical device 310). In some implementations, Figure 5 One or more processing blocks may be executed by one or more processors of the optical device (e.g., one or more processors 132 or one or more processors 320). In some embodiments, Figure 5 One or more process frames may be executed by another device, or a group of devices that are separate from or include one or more processors, such as user equipment (e.g., user equipment 340). Additionally or alternatively, Figure 5 One or more process frames may be executed by one or more components of the device 400, such as processor 420, memory 430, storage component 440, input component 450, output component 460 and / or communication component 470.

[0061] In some embodiments, the optical device may include, in addition to one or more processors, an optical sensor comprising one or more diffuse optical elements and an array of sensor elements disposed on the surface of the display screen. The one or more diffuse optical elements may be configured to diffuse and distribute a first portion of the light associated with the object across the array of sensor elements of the optical sensor. The display screen may include one or more transmissive elements that allow the first portion of the light to pass through the display screen via its surface; and one or more blocking elements that prevent a second portion of the light associated with the object from passing through the display screen.

[0062] like Figure 5 As shown, process 500 may include obtaining sensor data associated with a first portion of light from an optical sensor (block 510). For example, one or more processors may obtain sensor data associated with the first portion of light from an optical sensor, as described above.

[0063] like Figure 5 As further shown, process 500 may include processing sensor data to determine an image of an object (box 520). For example, one or more processors may process the sensor data to determine an image of the object, as described above.

[0064] like Figure 5 As further shown, process 500 may include performing one or more actions based on an image of an object (box 530). For example, one or more processors may perform one or more actions based on an image of an object, as described above.

[0065] Process 500 may include additional implementations, such as any single implementation or any combination of implementations of one or more other processes described below and / or elsewhere herein.

[0066] In a first embodiment, one or more transmissive components include one or more pixel components of the display screen.

[0067] In the second embodiment, one or more transmissive components, alone or in combination with the first embodiment, include one or more window components of the display screen, wherein the window components are located in the first pixel component and the second pixel component of the display screen.

[0068] In the third embodiment, alone or in combination with one or more of the first and second embodiments, one or more transmissive components include one or more gaps between the pixel components of the display screen.

[0069] In the fourth embodiment, alone or in combination with one or more of the first to third embodiments, one or more blocking components include one or more circuit components of the display screen.

[0070] In the fifth embodiment, alone or in combination with one or more of the first to fourth embodiments, processing sensor data to determine an image of an object includes: processing sensor data to identify a first set of sensor data associated with a first portion of light; processing sensor data to identify a second set of sensor data associated with light emitted by one or more pixel components of the display screen; and processing the first set of sensor data, but not the second set of sensor data, to determine an image of the subject.

[0071] In the sixth embodiment, processing sensor data to determine an image of an object, either alone or in combination with one or more of the first to fifth embodiments, includes: determining calibration information associated with the circuit architecture of the display screen, such that an algorithm for constructing an image based on light diffused by one or more diffuse optical elements is optimized according to the calibration information; and using the algorithm to process the sensor data to determine an image of the object.

[0072] In the seventh embodiment, processing sensor data to determine an image of an object, alone or in combination with one or more of the first to sixth embodiments, includes: processing sensor data using a machine learning model to determine an image of the object.

[0073] In the eighth embodiment, one or more actions are performed, either alone or in combination with one or more of the first to seventh embodiments, including retraining the machine learning model based on the image of the object.

[0074] In the ninth embodiment, alone or in combination with one or more of the first to eighth embodiments, the object is the face of a user of an optical device and performing one or more actions includes: processing an image of the object using facial recognition analysis.

[0075] In the tenth embodiment, performing one or more actions, alone or in combination with one or more of the first to ninth embodiments, includes displaying an image of the object on a display screen.

[0076] Although Figure 5 An example block diagram of process 500 is shown, but in some implementations, process 500 may include... Figure 5 The boxes described herein are those that are additional, fewer, different, or arranged differently. Alternatively, two or more boxes of process 500 may be executed in parallel.

[0077] Figure 6 This is a flowchart of an example process 600 associated with an optical device (e.g., optical device 310). In some embodiments, Figure 6One or more processing blocks may be executed by one or more processors of the optical device (e.g., one or more processors 132 or one or more processors 320). In some embodiments, Figure 6 One or more process frames may be executed by another device, or a group of devices that are separate from or include one or more processors, such as user equipment (e.g., user equipment 340). Additionally or alternatively, Figure 6 One or more process frames may be executed by one or more components of the device 400, such as processor 420, memory 430, storage component 440, input component 450, output component 460 and / or communication component 470.

[0078] like Figure 6 As shown, process 600 may include: obtaining sensor data from an optical sensor of an optical device associated with a portion of light, which is associated with an object and diffused and distributed across the input surface of the optical sensor by one or more diffuse optical elements disposed on the surface of the display screen (box 610). For example, one or more processors may obtain sensor data from an optical sensor of an optical device associated with a portion of light, which is associated with an object and diffused and distributed across the input surface of the optical sensor by one or more diffuse optical elements disposed on the surface of the display screen, as described above. In some embodiments, the display screen includes one or more transmissive components that allow a portion of light to pass through the display screen. In some embodiments, the display screen includes one or more blocking components that prevent different portions of light associated with an object from passing through the display screen.

[0079] like Figure 6 As further shown, process 600 may include processing sensor data to determine an image of an object (box 620). For example, one or more processors may process the sensor data to determine an image of the object, as described above.

[0080] like Figure 6 As further shown, process 600 may include providing an image of the object (box 630). For example, one or more processors may provide an image of the object, as described above.

[0081] Process 600 may include additional implementations, such as any single implementation or any combination of implementations of one or more other processes described below and / or elsewhere herein.

[0082] In a first embodiment, providing an image of an object includes displaying the image of the object on a display screen.

[0083] In the second embodiment, providing an image of an object, alone or in combination with the first embodiment, includes sending the image of the object to another device so that the other device processes the image of the object using facial recognition analysis.

[0084] In a third embodiment, processing sensor data to determine an image of an object, either alone or in combination with one or more of the first and second embodiments, includes: identifying an algorithm for constructing an image based on light diffused by one or more diffuse optical elements disposed on the surface of a display screen, and using the algorithm to process the sensor data to determine an image of the object.

[0085] In the fourth embodiment, processing sensor data to determine an image of an object, either alone or in combination with one or more of the first to third embodiments, includes: processing sensor data using a machine learning model to determine an image of the object.

[0086] Although Figure 6 An example block diagram of process 600 is shown, but in some implementations, process 600 may include... Figure 6 The boxes described herein are those that are additional, fewer, different, or arranged differently compared to other boxes. Alternatively, two or more boxes of process 600 may be executed in parallel.

[0087] Figure 7 This is a flowchart of an example process 700 associated with an optical device (e.g., optical device 310). In some embodiments, Figure 7 One or more process blocks can be executed by optical devices. In some implementations, Figure 7 One or more process frames may be performed by another device, or a group of devices separate from or including the optical device, such as user equipment (e.g., user equipment 340). Additionally or alternatively, Figure 7 One or more process frames may be executed by one or more components of the device 400, such as processor 420, memory 430, storage component 440, input component 450, output component 460 and / or communication component 470.

[0088] like Figure 7As shown, process 700 may include obtaining sensor data from an optical sensor of an optical device associated with a portion of light, which is associated with an object, transmitted via a display screen, and diffused and distributed in a caustic pattern across the input surface of the optical sensor by one or more diffuse optical elements (box 710). For example, an optical device may obtain sensor data from an optical sensor of an optical device associated with a portion of light, which is associated with an object, transmitted via a display screen, and diffused and distributed in a caustic pattern across the input surface of the optical sensor by one or more diffuse optical elements, as described above.

[0089] like Figure 7 As further shown, process 700 may include processing sensor data based on caustic patterns to determine an image of an object (box 720). For example, an optical device may process sensor data based on caustic patterns to determine an image of an object, as described above.

[0090] like Figure 7 As further shown, process 700 may include performing one or more actions based on an image of an object (box 730). For example, as described above, an optical device may perform one or more actions based on an image of an object.

[0091] Process 700 may include additional implementations, such as any single implementation or any combination of implementations of one or more other processes described below and / or elsewhere herein.

[0092] In a first embodiment, the display screen includes one or more transmissive components that allow partial light to pass through the display screen, and one or more blocking components that prevent different portions of light associated with an object from passing through the display screen.

[0093] In the second embodiment, processing sensor data to determine an image of an object, either alone or in combination with the first embodiment, includes: processing sensor data using an algorithm to determine at least one of the following: the algorithm being configured to construct an image based on a caustic pattern; or processing sensor data using a machine learning model to determine an image of an object based on a caustic pattern.

[0094] In the third embodiment, one or more actions are performed, either alone or in combination with one or more of the first and second embodiments, including at least one of the following: displaying an image of the object on a display screen, processing the image of the object using facial recognition analysis technology, or retraining a machine learning model based on the image of the object, wherein the machine learning model is used to process sensor data to determine the image of the object.

[0095] Although Figure 7 An example block of process 700 is shown, but in some implementations, process 700 may include... Figure 7 The boxes described herein are those that are additional, fewer, different, or arranged differently compared to other boxes. Alternatively, two or more boxes of process 700 may be executed in parallel.

[0096] The foregoing disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Modifications and variations can be made based on the foregoing disclosure, or these modifications and variations can be obtained from practice of the embodiments.

[0097] As used herein, the term "component" is intended to be interpreted broadly as hardware, firmware, or a combination of hardware and software. Clearly, the systems and / or methods described herein can be implemented in various forms of hardware, firmware, and / or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to any specific software code, and it should be understood that the systems and / or methods can be implemented using software and hardware based on the description herein.

[0098] Even if a specific combination of features is stated in the claims and / or disclosed in the specification, such combinations are not intended to limit the disclosure of various embodiments. In fact, many of these features can be combined in ways not specifically stated in the claims and / or not specifically disclosed in the specification. Although each dependent claim listed below may be directly dependent on only one claim, the disclosure of various embodiments includes combinations of each dependent claim with every other claim in the claim set. As used herein, the phrase “at least one” in the list of items refers to any combination of those items, including a single member. As an example, “at least one of a, b, or c” is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical items.

[0099] Unless explicitly stated otherwise, no element, action, or instruction used herein should be construed as critical or necessary. Furthermore, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more.” Furthermore, as used herein, the term “described” is intended to include one or more items associated with the term “described” and may be used interchangeably with “described one or more.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, or a combination of related and unrelated items) and may be used interchangeably with “one or more.” If only one item is intended, the phrase “only one item” or similar language is used. Furthermore, as used herein, the terms “have,” “contain,” “own,” etc., are intended to be open-ended terms. Furthermore, unless explicitly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.” Furthermore, as used herein, unless explicitly stated otherwise (e.g., if used in conjunction with “any” or “only one of them”), the term “or” when used in series is intended to be inclusive and may be used interchangeably with “and / or.”

Claims

1. An optical device, comprising: Optical sensors, including sensor element arrays; and One or more diffuse optical elements are disposed on the surface of the display screen and configured to diffuse and distribute a first portion of the light associated with the object across the sensor element array of the optical sensor. The display screen includes one or more transmissive elements that allow a first portion of the light to pass through the surface of the display screen. The display screen includes one or more blocking components that prevent a second portion of the light associated with the object from passing through the display screen; The one or more blocking components include one or more circuit components of the display screen.

2. The optical device of claim 1, wherein the one or more transmission components include one or more pixel components of the display screen.

3. The optical device of claim 1, wherein the one or more transmission components include one or more window components of the display screen. The window component is positioned between the first pixel component and the second pixel component of the display screen.

4. The optical device of claim 1, wherein the one or more transmission components include one or more gaps between the pixel components of the display screen.

5. The optical device according to claim 1, further comprising one or more processors, The one or more processors mentioned above are configured to: Sensor data associated with a first portion of the light is obtained from the optical sensor; Process the sensor data to determine an image of the object; and Perform one or more actions based on the image of the object.

6. The optical device of claim 5, wherein the one or more processors are configured to, when processing the sensor data to determine an image of the object: The sensor data is processed to identify a first set of sensor data associated with a first portion of the light; The sensor data is processed to identify a second set of sensor data that is associated with light emitted by one or more pixel components of the display screen; and The first set of sensor data, rather than the second set of sensor data, is processed to determine an image of the object.

7. The optical device of claim 5, wherein the one or more processors are configured to, when processing the sensor data to determine an image of the object: Determine the calibration information associated with the circuit architecture of the display screen; Based on the calibration information, the algorithm for constructing an image based on light diffused by the one or more diffuse optical elements is optimized; as well as The algorithm is used to process the sensor data to determine an image of the object.

8. The optical device of claim 5, wherein the one or more processors are configured to, when processing the sensor data to determine an image of the object: The sensor data is processed using a machine learning model to determine an image of the object.

9. The optical device of claim 5, wherein the object is the face of the user of the optical device. The one or more processors are configured to, when performing the one or more actions: The image of the object is processed using facial recognition analysis.

10. The optical device of claim 5, wherein the one or more processors are configured to, when performing the one or more actions: The image of the object is displayed on the display screen.

11. A non-transitory computer-readable medium storing instructions, the instructions comprising: One or more instructions, when executed by one or more processors of the optical device, cause the one or more processors to: Sensor data associated with a portion of light is obtained from the optical sensor of the optical device. This portion of light is associated with an object and is diffused and distributed across the input surface of the optical sensor by one or more diffuse optical elements disposed on the surface of the display screen. The display screen includes one or more transmissive elements that allow a portion of the light to pass through the display screen. The display screen includes one or more blocking components that prevent different portions of light associated with the object from passing through the display screen; The one or more blocking components include one or more circuit components of the display screen; Process the sensor data to determine an image of the object; and Provide an image of the object.

12. The non-transitory computer-readable medium of claim 11, wherein the one or more instructions causing the optical device to provide an image of the object cause the optical device to: The image of the object is displayed on the display screen.

13. The non-transitory computer-readable medium of claim 11, wherein the one or more instructions that cause the optical device to provide an image of the object cause the optical device to: The image of the object is sent to another device so that the other device can process the image of the object using facial recognition analysis.

14. The non-transitory computer-readable medium of claim 11, wherein the one or more instructions for causing the optical device to process the sensor data to determine an image of the object cause the optical device to: An algorithm that identifies an image based on light diffused by one or more diffuse optical elements disposed on the surface of the display screen; and The algorithm is used to process the sensor data to determine an image of the object.

15. The non-transitory computer-readable medium of claim 11, wherein the one or more instructions for causing the optical device to process the sensor data to determine an image of the object cause the optical device to: The sensor data is processed using a machine learning model to determine an image of the object.

16. A method comprising: Sensor data associated with a portion of light, which is associated with an object, is obtained from an optical device and from the optical sensor of the optical device, is transmitted through a display screen, and is diffused and distributed in a caustic pattern across the input surface of the optical sensor by one or more diffuse optical elements; The sensor data is processed by the optical device and based on the caustic pattern to determine an image of the object; as well as The optical device performs one or more actions based on an image of the object; The display screen includes one or more transmissive elements and one or more blocking elements, the transmissive elements allowing a portion of light to pass through the display screen, and the blocking elements preventing different portions of light associated with the object from passing through the display screen; as well as The one or more blocking components include one or more circuit components of the display screen.

17. The method of claim 16, wherein processing the sensor data to determine an image of the object comprises at least one of the following: The sensor data is processed using an algorithm to determine an image of the object, wherein the algorithm is configured to construct an image based on the caustic pattern; or The sensor data is processed using a machine learning model to determine an image of the object based on the caustic pattern.

18. The method of claim 16, wherein performing the one or more actions comprises at least one of the following: Display an image of the object on the display screen; The image of the object is processed using facial recognition analysis technology; or The machine learning model is retrained based on the image of the object, wherein the machine learning model is used to process the sensor data to determine the image of the object.

Images