Image processing method, device and system and storage medium

Abstract

The invention provides an image processing method, device and system and a storage medium, the image processing method is applied to the image processing system, and the image processing system comprises a visual imaging module and a data processing module; the visual imaging module is connected with the data processing module; the visual imaging module comprises a zoom lens group, a micro-nano optical device and an image sensor which are sequentially adjacent in the optical axis direction; the image processing method comprises the following steps: determining an image of a target object through a visual imaging module; the image of the target object is obtained by transmitting light rays emitted by the target object or reflected light rays to the image sensor after the light rays are acted by the zoom lens group and the micro-nano optical device; performing data analysis on the image of the target object through a data processing module to obtain optical information of light of the target object; the optical information comprises at least two of the light intensity, the spectrum, the polarization and the direction of the light, so that the convenience of acquiring the multi-dimensional optical information of the light of the target object by the image processing system is improved.

Description

Image processing methods, apparatus, systems and storage media Technical Field

[0001] This application relates to the field of imaging technology, and in particular to an image processing method, apparatus, system and storage medium. Background Technology

[0002] In related technologies, image sensors can include RGB sensors, spectral sensors, polarization sensors, event sensors, and so on. However, the aforementioned sensors can only collect optical information of light in a single dimension, resulting in the loss of optical information of light in other dimensions. This can easily lead to poor ease of acquiring multi-dimensional optical information of light. Summary of the Invention

[0003] This application provides an image processing method, apparatus, system, and storage medium, aiming to improve the ease of acquiring multi-dimensional optical information of light from a target object in an image processing system.

[0004] In a first aspect, this application provides an image processing method applied to an image processing system, the image processing system including a visual imaging module and a data processing module; the visual imaging module is connected to the data processing module; the visual imaging module includes a zoom lens group, micro / nano optical devices, and an image sensor sequentially adjacent in the optical axis direction; the image processing method includes: generating an image of the target object when light emitted or reflected from a target object passes through the visual imaging module; and performing data analysis on the image of the target object through the data processing module to obtain optical information of the light emitted by the target object; the optical information includes at least two of the following: light intensity, spectrum, polarization, and direction.

[0005] Secondly, this application provides an image processing apparatus, comprising: an image acquisition module, configured to generate an image of the target object when light emitted or reflected from the target object passes through the visual imaging module of an image processing system; and a data analysis module, configured to perform data analysis on the image of the target object through the data processing module of the image processing system to obtain optical information of the light emitted from the target object; wherein the optical information includes at least two of the following: light intensity, spectrum, polarization, and direction.

[0006] Thirdly, this application provides an image processing system, which includes a visual imaging module and a data processing module; the visual imaging module is connected to the data processing module; the visual imaging module includes a zoom lens group, a micro / nano optical device, and an image sensor that are sequentially adjacent in the optical axis direction; the zoom lens group is used to change the imaging focal length of the zoom lens group; the micro / nano optical device is used to collect and / or adjust the optical information of light; the light emitted or reflected by the target object is transmitted to the image sensor after being acted upon by the zoom lens group and the micro / nano optical device to obtain an image of the target object; the data processing module is used to acquire the image of the target object determined by the visual imaging module, and to perform data analysis on the image of the target object to obtain the optical information of the light of the target object; the optical information includes at least two of the light intensity, spectrum, polarization, and direction of the light of the target object.

[0007] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the image processing method described above.

[0008] This application provides an image processing method, apparatus, system, and storage medium. In an image processing system comprising a visual imaging module, and the visual imaging module including a zoom lens group, micro / nano optical devices, and an image sensor sequentially adjacent along the optical axis, the zoom lens group is used to change the imaging focal length of the zoom lens group. Therefore, the image processing system can acquire light rays emitted or reflected from or by a target object at multiple incident angles through the zoom lens group of the visual imaging module. Under the action of the zoom lens group, the image of the target object can be used by the data processing module included in the subsequent image processing system to determine the direction of the light rays from the target object. Furthermore, since the micro / nano optical devices are used to collect and / or adjust the optical information of the light rays, the image processing system can collect and / or adjust the optical information of the light rays emitted or reflected from the target object through the micro / nano optical devices of the visual imaging module, so that the image sensor can determine the image of the target object. Under the action of the micro / nano optical devices, the image of the target object can also be used by the subsequent data processing module to determine at least one of the light intensity, spectrum, and polarization of the light rays from the target object. Based on this, the image processing system can use the visual imaging module to determine the image of the target object, and use the data processing module to determine the multi-dimensional optical information of the target object's light, which helps to improve the ease with which the image processing system can acquire the multi-dimensional optical information of the target object's light. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 is a schematic block diagram of the structure of an image processing system provided in an embodiment of this application; Figure 2 is a schematic diagram of the structure of a visual imaging module included in an image processing system according to an embodiment of this application; Figure 3 is a schematic diagram of the light transmission path of the visual imaging module included in an image processing system according to an embodiment of this application; Figure 4 is a schematic diagram of image processing in an image processing system according to an embodiment of this application; Figure 5 is a schematic diagram of the structure of micro-nano optical devices included in an image processing system according to an embodiment of this application; Figure 6 is a schematic diagram of an imaging of a zoom lens group included in an image processing system according to an embodiment of this application; Figure 7 is another schematic diagram of an imaging of a zoom lens group included in an image processing system according to an embodiment of this application; Figure 8 is a flowchart of an image processing method provided in an embodiment of this application; Figure 9 is a schematic block diagram of an image processing device provided in an embodiment of this application; Figure 10 is a schematic block diagram of the structure of an image processing system provided in yet another embodiment of this application.

[0011] Explanation of reference numerals in the attached figures: 10, image processing system; 100, visual imaging module; 110, zoom lens group; 111, zoom device; 112, imaging lens group; 120, micro / nano optical device; 121, first micro / nano optical device; 1211, microlens; 122, second micro / nano optical device; 130, image sensor; 131, detection pixel; 200, data processing module. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0014] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0015] Please refer to Figures 1 and 2. Figure 1 is a schematic block diagram of the structure of an image processing system 10 provided in an embodiment of this application. Figure 2 is a schematic diagram of the structure of a visual imaging module 100 included in the image processing system 10 according to an embodiment of this application.

[0016] As shown in Figures 1 and 2, the image processing system 10 includes a visual imaging module 100 and a data processing module 200; the visual imaging module 100 is connected to the data processing module 200.

[0017] The visual imaging module 100 includes a zoom lens group 110, a micro-nano optical device 120, and an image sensor 130 that are sequentially adjacent in the optical axis direction; the zoom lens group 110 is used to change the imaging focal length of the zoom lens group 110; the micro-nano optical device 120 is used to collect and / or adjust the optical information of the light.

[0018] The light emitted or reflected by the target object is transmitted to the image sensor 130 after passing through the zoom lens group 110 and the micro-nano optical device 120, thus obtaining an image of the target object.

[0019] The data processing module 200 is used to acquire an image of the target object determined by the visual imaging module 100, and to perform data analysis on the image of the target object to obtain the optical information of the light rays of the target object; the optical information includes at least two of the following: light intensity, spectrum, polarization, and direction of the light rays of the target object.

[0020] For example, the image processing system 10 includes a visual imaging module 100 and a data processing module 200, with the visual imaging module 100 connected to the data processing module 200. For instance, the visual imaging module 100 can be connected to the data processing module 200 via a wired connection. The visual imaging module 100 can also be connected to the data processing module 200 via a wireless connection. For example, the visual imaging module 100 and the data processing module 200 may be communicatively connected. However, this is not a limitation and is not intended to restrict the scope of the system.

[0021] For example, in the initial state of the image processing system 10, the zoom lens group 110 included in the visual imaging module 100 has an imaging focal length greater than or equal to 13mm and less than or equal to 120mm. In the initial state, the imaging focal length of the zoom lens group 110 can cover macro, wide-angle, main camera, telephoto, etc. When the zoom lens group 110 is used to change its imaging focal length, the imaging focal length of the zoom lens group 110 can be changed. For example, the imaging focal length of the zoom lens group 110 can be changed from greater than or equal to 13mm and less than or equal to 120mm to greater than or equal to 1mm and less than or equal to 200mm. Of course, the imaging focal length of the zoom lens group 110 and the ways in which its imaging focal length can be changed are not limited to this, and are not restricted here.

[0022] For example, the image sensor 130 includes one of a visible light sensor, an event sensor, a near-infrared sensor, a mid-infrared sensor, and a far-infrared sensor. The image sensor 130 can acquire sensor information in different spectral bands, thereby improving the flexibility of the image processing system 10 in acquiring optical information about the light of the target object. Of course, the image sensor 130 is not limited to this, and no limitation is made here.

[0023] For example, the target object may include objects in the target scene where the visual imaging module 100 is located. The target object may include one or more of the environment, objects, people, etc., without limitation. Accordingly, the number of target objects may include at least one, without limitation. For example, in the process of capturing the target scene to determine the image of the target scene, the visual imaging module 100 may collect the light emitted or reflected by the target objects included in the target scene to form an image of the target objects included in the target scene, thereby forming an image of the target scene.

[0024] When the visual imaging module 100 includes a zoom lens group 110, a micro-nano optical device 120 and an image sensor 130 that are sequentially adjacent in the optical axis direction, the light emitted or reflected by the target object can be transmitted to the image sensor 130 after being acted upon by the zoom lens group 110 and the micro-nano optical device 120 in sequence, so that the image sensor 130 can determine the image of the target object.

[0025] For example, when the zoom lens group 110 is used to change the imaging focal length of the zoom lens group 110, if the imaging focal length of the zoom lens group 110 changes, the viewing angle of the visual imaging module 100 where the zoom lens group 110 is located will change accordingly. The viewing angle of the visual imaging module 100 can be used to determine the range of incident angles of the light rays that the visual imaging module 100 can collect and use for imaging; therefore, the viewing angle of the visual imaging module 100 can also be referred to as the incident angle range corresponding to the visual imaging module 100. When the incident angle of the light rays emitted or reflected by the target object into the visual imaging module 100 is within the incident angle range corresponding to the visual imaging module 100, the visual imaging module 100 can collect the light rays emitted or reflected by the target object and perform imaging processing on the light rays emitted or reflected by the target object to determine the image of the target object. The image of the target object can be used by the image processing system 10 to subsequently determine the optical information of the light rays of the target object, such as determining the direction of the light rays of the target object. For example, the image processing system 10 can perform data analysis on the image of the target object determined by the visual imaging module 100 through the data processing module 200 to obtain the incident angle of the light rays from the target object when they are incident on the visual imaging module 100, and then determine the direction of the light rays from the target object based on the incident angle. Of course, it is not limited to this, and no limitation is made here.

[0026] In some implementations, when the target scene includes multiple target objects, the object distances of the different target objects can be different. The visual imaging module 100 can have a specific depth of field range. If the object distance of the target object is within the depth of field range of the visual imaging module 100, the visual imaging module 100 can acquire a clear image of the target object. If the object distance of the target object is outside the depth of field range of the visual imaging module 100, the visual imaging module 100 can only acquire a blurry image of the target object. When the visual imaging module 100 includes a zoom lens group 110, the visual imaging module 100 can change the depth of field range of the visual imaging module 100 by changing the imaging focal length of the zoom lens group 110, thereby ensuring that target objects at different object distances are all within the depth of field range of the visual imaging module 100, thus acquiring a clear image of the target object. For example, during the process of changing the imaging focal length of the zoom lens group 110, the visual imaging module 100 can be linked to focus to dynamically adjust the depth-of-field range of the visual imaging module 100. This ensures that target objects at different object distances can fall within the depth-of-field range of the visual imaging module 100, meaning that target objects at different object distances can all fall within the sharp imaging range of the visual imaging module 100, resulting in a clear image of the target object. Once a clear image of the target object is obtained, the image processing system 10 can perform data analysis on the image of the target object through the data processing module 200 to determine the optical information of the target object's light rays, such as determining the direction of the target object's light rays. However, this is not a limitation and is not set forth here.

[0027] For example, the visual imaging module 100 can change the imaging focal length of the zoom lens group 110 based on the object distance of the target object. In an exemplary embodiment, the visual imaging module 100 can acquire the object distance of the target object. When the visual imaging module 100 acquires the object distance of the target object, it can determine the required imaging focal length of the zoom lens group 110 when the visual imaging module 100 acquires a clear image of the target object based on that object distance. When the current imaging focal length of the zoom lens group 110 is inconsistent with the required imaging focal length of the zoom lens group 110 when the visual imaging module 100 acquires a clear image of the target object, the imaging focal length of the zoom lens group 110 can be changed based on the required imaging focal length of the zoom lens group 110 when the visual imaging module 100 acquires a clear image of the target object. When the current imaging focal length of the zoom lens group 110 is consistent with the required imaging focal length of the zoom lens group 110 when the visual imaging module 100 acquires a clear image of the target object, it can be determined that there is no need to change the imaging focal length of the zoom lens group 110. Based on this, the light emitted or reflected from the target object can be imaged onto the micro / nano optical device 120 by either the zoom lens group 110 with or without a changed focal length, and then transmitted to the image sensor 130 after being processed by the micro / nano optical device 120, resulting in a clear image of the target object. However, this is not a limitation and is not set forth herein.

[0028] For example, the optical information of light can include at least one of the following: polarization, spectrum, intensity, etc. As shown in Figure 3, when light emitted or reflected from the target object is transmitted to the micro / nano optical device 120 through the zoom lens group 110, the micro / nano optical device 120 can collect and / or adjust the optical information of the light. In an exemplary embodiment, the micro / nano optical device 120 can achieve the collection and / or adjustment of optical information such as polarization, spectrum, and intensity of light through subwavelength-scale micro / nano structure design. For example, the micro / nano optical device 120 can utilize anisotropic micro / nano structures to collect and / or adjust the polarization of light. For example, the micro / nano optical device 120 can utilize the resonance effect in the micro / nano structure to collect and / or adjust the spectrum of light. For example, the micro / nano optical device 120 can utilize the tunable absorption, reflection, or transmittance of the micro / nano structure to collect and / or adjust the intensity of light. Of course, it is not limited to these, and no limitation is made here.

[0029] For example, the micro / nano optical device 120 may include at least one of a microlens 1211, a microlens array, a metasurface, etc., without limitation. The micro / nano optical device 120 can be used to acquire and / or adjust at least one of the optical information such as polarization, spectrum, and intensity of light. For example, the micro / nano optical device 120 may be designed to acquire and / or adjust one of the optical information such as spectrum, polarization, and intensity, or it may be designed to acquire and / or adjust at least two of the optical information such as spectrum, polarization, and intensity, without limitation. After the micro / nano optical device 120 acquires and / or adjusts the optical information of the light, the light after being processed by the micro / nano optical device 120 can continue to be transmitted to the image sensor 130 for the image sensor 130 to determine the image of the target object. The image of the target object can be used by the data processing module 200 included in the image processing system 10 to determine the optical information of the light of the target object, such as determining at least one of the optical information such as spectrum, polarization, and intensity of the light of the target object. For example, when an image of a target object is determined, the image processing system 10 can perform data analysis on the image of the target object through the data processing module 200 to determine the optical information of the target object's light, such as the spectrum, polarization, and intensity of the light. Of course, it is not limited to this, and no limitation is made here.

[0030] As shown in Figure 4, the image of the target object determined by the visual imaging module 100 can be analyzed by the data processing module 200 to obtain multi-dimensional optical information of the light emitted or reflected by the target object. For example, the data processing module 200 can process the image of the target object provided by the visual imaging module 100 based on at least one of the following methods: Neural Processing Unit (NPU), Application-Specific Integrated Circuit (ASIC), etc., to obtain the optical information of the light emitted by the target object. The optical information of the light emitted by the target object can include at least two of the following: light intensity, spectrum, polarization, and direction of the light emitted by the target object, without limitation. The data processing module 200 can have at least one of the following data processing functions: numerical analysis function, deep learning function, etc., and thus, the data processing module 200 can simultaneously analyze and determine the multi-dimensional optical information of the light emitted by the target object by performing data analysis on the image of the target object determined by the visual imaging module 100. For example, the data processing module 200 can include a multimodal large model. The multimodal large model can be trained based on images of different preset objects determined by the visual imaging module 100. Different preset objects can include the target object as well as other objects besides the target object. Of course, they are not limited to this, and no restrictions are imposed here.

[0031] For example, multi-dimensional optical information can include at least two of the following: polarization, spectrum, intensity, and direction of light. For instance, the image processing system 10 can use the data processing module 200 in conjunction with the image of the target object determined by the visual imaging module 100 to simultaneously analyze and determine at least two of the optical information of the target object, such as polarization, spectrum, intensity, and direction of light. Of course, multi-dimensional optical information is not limited to this; for example, the data processing module 200 can also determine the depth of field corresponding to the target object based on the image of the target object, without further limitation.

[0032] When the image processing system 10 determines the multi-dimensional optical information of the light rays of the target object, such as at least two of the polarization, spectrum, intensity, and direction of the light rays of the target object, the multi-dimensional optical information of the light rays of the target object can be used for multi-modal object detection, strong reflection removal in photography, acquisition of multi-dimensional incident angles, etc., which is conducive to improving the image processing system 10's ability to perceive the real target scene in which the target object is located.

[0033] For example, the image processing system 10, including a visual imaging module 100 and a data processing module 200, can be distributed across different electronic devices, with the visual imaging module 100 and the data processing module 200 communicatively connected. Taking an example where the visual imaging module 100 is located on electronic device 1 and the data processing module 200 is located on electronic device 2, when the visual imaging module 100 acquires an image of a target object, it can transmit the image of the target object to the data processing module 200 based on the communication connection between the visual imaging module 100 and the data processing module 200. This allows the data processing module 200 to determine the multi-dimensional optical information of the target object's light based on the image, thereby perceiving the real target scene in which the target object is located. Of course, it is not limited to this. The visual imaging module 100 and the data processing module 200 included in the image processing system 10 can also be set on the same electronic device. For example, the image processing system 10 can be applied to different types of electronic devices to reduce the number of image sensors 130 required on different types of electronic devices, reduce costs, reduce power consumption, reduce size, and at the same time improve the ease of acquisition and analysis of multi-dimensional optical information of the target object, so that the electronic device can perceive the real target scene where the target object is located. No restrictions are imposed here.

[0034] Based on this, when the image processing system 10 includes a visual imaging module 100, and the visual imaging module 100 includes a zoom lens group 110, a micro-nano optical device 120, and an image sensor 130 that are sequentially adjacent in the optical axis direction, the zoom lens group 110 and the micro-nano optical device 120 included in the visual imaging module 100 can be designed to align with the image sensor 130. Since the zoom lens group 110 is used to change the imaging focal length of the zoom lens group 110, the image processing system 10 can acquire light rays emitted or reflected from or from the target object at multiple incident angles through the zoom lens group 110 of the visual imaging module 100. Under the action of the zoom lens group 110, the image of the target object can be used by the data processing module 200 included in the subsequent image processing system 10 to determine the direction of the light rays from the target object. Furthermore, since the micro-nano optical device 120 is used to collect and / or adjust the optical information of light, the image processing system 10 can collect and / or adjust the optical information of the light emitted or reflected by the target object through the micro-nano optical device 120 of the visual imaging module 100, so that the image sensor 130 can determine the image of the target object. Under the action of the micro-nano optical device 120, the image of the target object can also be used by the subsequent data processing module 200 to determine at least one of the light intensity, spectrum, and polarization of the light of the target object. Compared with a single image sensor 130, which can only acquire optical information in a single dimension, the image of the target object acquired by the visual imaging module 100 in the image processing system 10 can be used by the data processing module 200 to determine the multi-dimensional optical information of the light of the target object, which is beneficial to improving the convenience of the image processing system 10 in acquiring the multi-dimensional optical information of the light of the target object.

[0035] In the image processing system 10, the visual imaging module 100 can work in conjunction with the zoom lens group 110, micro / nano optical devices 120, and image sensor 130 to determine the image of the target object. Furthermore, the data processing module 200 in the image processing system 10 can utilize the image of the target object determined by the visual imaging module 100 to determine the optical information of the target object's light rays, such as at least two of the light intensity, spectrum, polarization, and direction of the target object's light rays. In this case, the image processing system 10 does not need to include multiple different types of image sensors 130 in the visual imaging module 100 to separately acquire the optical information of the target object's light rays in different dimensions using different types of image sensors 130. For example, the visual imaging module 100 of the image processing system 10 can also be referred to as a single-mono multimodal sensor, indicating that the visual imaging module 100 can utilize a single type of image sensor 130 to acquire multidimensional optical information of the target object's light rays. Accordingly, when the visual imaging module 100 can use a single type of image sensor 130 to acquire multi-dimensional optical information of the target object's light, it can avoid the situation where the data acquired by different types of image sensors 130 cannot achieve pixel-level alignment due to parallax between the data acquired by each type of image sensor 130. Based on the design of the visual imaging module 100 in the image processing system 10, it is beneficial to reduce the number of image sensors 130 included in the image processing system 10, thereby reducing the size of the image processing system 10, and lowering the power consumption and cost of the image processing system 10. Accordingly, based on the design of the visual imaging module 100 and the data processing module 200 in the image processing system 10, it is beneficial to improve the ease with which the image processing system 10 acquires multi-dimensional optical information of the target object's light.

[0036] In one embodiment, the micro-nano optical device 120 includes a first micro-nano optical device 121 and a second micro-nano optical device 122; in the optical axis direction, the first micro-nano optical device 121 is located between the zoom lens group 110 and the second micro-nano optical device 122; or, in the optical axis direction, the second micro-nano optical device 122 is located between the zoom lens group 110 and the first micro-nano optical device 121.

[0037] For example, the first micro-nano optical device 121 and the second micro-nano optical device 122 can be of the same type as the micro-nano optical device 120, or they can be of different types. For instance, both the first micro-nano optical device 121 and the second micro-nano optical device 122 can be metasurfaces. Of course, this is not a limitation; the first micro-nano optical device 121 can be a microlens 1211 or a microlens array, and the second micro-nano optical device 122 can be a metasurface; no limitation is imposed here.

[0038] The positions of the first micro-nano optical device 121 and the second micro-nano optical device 122 can be interchanged. For example, in the optical axis direction, the first micro-nano optical device 121 is located between the zoom lens group 110 and the second micro-nano optical device 122; or, in the optical axis direction, the second micro-nano optical device 122 is located between the zoom lens group 110 and the first micro-nano optical device 121. Thus, when the micro-nano optical device 120 includes both the first micro-nano optical device 121 and the second micro-nano optical device 122, it is beneficial to improve the flexibility of the visual imaging module 100 in setting up the micro-nano optical device 120.

[0039] When the micro-nano optical device 120 includes a first micro-nano optical device 121 and a second micro-nano optical device 122, the image processing system 10 can simultaneously acquire and / or adjust the multi-dimensional optical information of the light from the target object using the first micro-nano optical device 121 and the second micro-nano optical device 122 of the visual imaging module 100. For example, the first micro-nano optical device 121 can be used to acquire and / or adjust the polarization of the light from the target object, and the second micro-nano optical device 122 can be used to acquire and / or adjust the spectrum of the light from the target object. However, this is not a limitation; the image processing system 10 can also simultaneously acquire and / or adjust the multi-dimensional optical information of the light from the target object using either the first micro-nano optical device 121 or the second micro-nano optical device 122 of the visual imaging module 100. Thus, when the micro-nano optical device 120 includes both the first micro-nano optical device 121 and the second micro-nano optical device 122, it is beneficial to improve the ease of acquisition and adjustment of the multi-dimensional optical information of the light from the target object by the image processing system 10.

[0040] In one embodiment, the distance between the first micro-nano optical device 121 and the second micro-nano optical device 122 in the optical axis direction is less than or equal to 1 mm.

[0041] For example, the distance between the first micro-nano optical device 121 and the second micro-nano optical device 122 in the optical axis direction may include one of 0.1 mm, 0.2 mm, 0.5 mm, 0.7 mm, and 1 mm. Of course, it is not limited to this and is not restricted here.

[0042] In another embodiment, the distance between the first micro-nano optical device 121 and the second micro-nano optical device 122 in the optical axis direction is greater than or equal to 1 mm and less than or equal to 3 mm.

[0043] For example, the distance between the first micro-nano optical device 121 and the second micro-nano optical device 122 along the optical axis can be one of 1mm, 1.3mm, 1.6mm, 1.9mm, 2.4mm, 2.8mm, or 3mm. Of course, it is not limited to this and is not restricted here.

[0044] When a certain distance exists between the first micro-nano optical device 121 and the second micro-nano optical device 122, if the first micro-nano optical device 121 is located between the zoom lens group 110 and the second micro-nano optical device 122, it can be ensured that the light from the target object is transmitted to the second micro-nano optical device 122 after being acted upon by the first micro-nano optical device 121, and then transmitted to the image sensor 130 after being acted upon by the second micro-nano optical device 122. Correspondingly, if the second micro-nano optical device 122 is located between the zoom lens group 110 and the first micro-nano optical device 121, it can be ensured that the light from the target object is transmitted to the first micro-nano optical device 121 after being acted upon by the second micro-nano optical device 122, and then transmitted to the image sensor 130 after being acted upon by the first micro-nano optical device 121.

[0045] Of course, it is not limited to this. When there is a corresponding distance between the first micro-nano optical device 121 and the second micro-nano optical device 122, it can be ensured that the necessary assembly structure gap is reserved between the first micro-nano optical device 121 and the second micro-nano optical device 122, which is conducive to the ease of assembly of the first micro-nano optical device 121 and the second micro-nano optical device 122 by the visual imaging module 100.

[0046] In one embodiment, the first micro-nano optical device 121 includes a plurality of microlenses 1211.

[0047] As shown in Figure 5, multiple microlenses 1211 can be arranged in a specific and regular manner on the plane in which they are located, thereby forming a microlens array. Each microlens 1211 in the microlens array can serve as a macropixel unit for acquiring and / or adjusting the optical information of the light from the target object.

[0048] Of course, the first micro-nano optical device 121 is not limited to this. The first micro-nano optical device 121 may also include multiple micro-nano optical devices 120 that are equivalent to the microlens 1211, such as metasurfaces, etc., without limitation.

[0049] In this way, the visual imaging module 100 can collect and / or adjust the optical information of the light of the target object through the multiple microlenses 1211 included in the first micro-nano optical device 121, which is conducive to improving the convenience of the visual imaging module 100 in collecting multi-dimensional optical information of the light of the target object.

[0050] In one embodiment, in the optical axis direction, the projection of each microlens 1211 onto the image sensor 130 is used to cover a predetermined number of detector pixels 131 included in the image sensor 130.

[0051] As shown in Figure 5, the projection of each microlens 1211 onto the image sensor 130 along the optical axis can be considered as the equivalent region of the corresponding microlens 1211 on the image sensor 130. The equivalent region of the microlens 1211 covers a predetermined number of detector pixels 131 included in the image sensor 130. For example, if the predetermined number includes at least two, then the projection of each microlens 1211 onto the image sensor 130 can cover at least two corresponding detector pixels 131. The equivalent regions of different microlenses 1211 on the image sensor 130 may overlap or may not overlap. For example, among the detector pixels 131 covered by different microlenses 1211, there may be at least one identical detector pixel 131, or there may be no identical detector pixels 131; this is not a limitation.

[0052] When the projection of the microlens 1211 onto the image sensor 130 along the optical axis covers a predetermined number of detector pixels 131 included in the image sensor 130, the visual imaging module 100 can work in conjunction with the microlens 1211 and the detector pixels 131 covered by the microlens 1211 to acquire optical information of the light from the target object. For example, the image processing system 10 can use the microlens 1211 of the visual imaging module 100 to focus light emitted or reflected from the target object at different incident angles, so that light at different incident angles converges onto different detector pixels 131 covered by the microlens 1211. This allows different detector pixels 131 to acquire optical information of light at different incident angles, such as the direction of light at different incident angles. Based on this, the image processing system 10 can work in conjunction with the microlens 1211 of the visual imaging module 100 and the detector pixels 131 covered by the microlens 1211 to improve the ease of acquiring multi-dimensional optical information of the light from the target object.

[0053] In one embodiment, the projection of the microlens 1211 onto the image sensor 130 is used to cover a first number of detector pixels 131 in a first direction; the projection of the microlens 1211 onto the image sensor 130 is used to cover a second number of detector pixels 131 in a second direction; wherein the first direction is perpendicular to the second direction, and both the first direction and the second direction are perpendicular to the optical axis direction.

[0054] For example, the first direction may include the horizontal direction of the plane where the image sensor 130 is located, and the second direction may include the vertical direction of the plane where the image sensor 130 is located. Both the horizontal and vertical directions of the plane where the image sensor 130 is located are perpendicular to the optical axis. Of course, it is not limited to this. The first direction may also include the vertical direction of the plane where the image sensor 130 is located, and the second direction may also include the horizontal direction of the plane where the image sensor 130 is located. There is no limitation here.

[0055] For example, the first number is greater than or equal to 2. The first number of detector pixels 131 covered by the projection of microlens 1211 onto image sensor 130 in the first direction may include one of 2, 3, 4, 5, 6, 7, 8, 9, and 10. For example, the second number is greater than or equal to 2. The second number of detector pixels 131 covered by the projection of microlens 1211 onto image sensor 130 in the second direction may include one of 2, 3, 4, 5, 6, 7, 8, 9, and 10. Of course, the first and second numbers are not limited to these and are not restricted here. Accordingly, the preset number of detector pixels 131 of image sensor 130 covered by the projection of microlens 1211 onto image sensor 130 may be determined based on the product of the first number and the second number.

[0056] As shown in Figure 5, the projection of the microlens 1211 onto the image sensor 130 covers a first number of detection pixels 131 in the first direction of 3. Correspondingly, if the projection of the microlens 1211 onto the image sensor 130 covers a second number of detection pixels 131 in the second direction of 3, then in the optical axis direction, the projection of the microlens 1211 onto the image sensor 130 covers the area corresponding to 3×3 detection pixels 131. In this case, the preset number of detection pixels 131 covered by the projection of the microlens 1211 onto the image sensor 130 can be 9.

[0057] For example, if the projection of microlens 1211 onto image sensor 130 covers a first number of 5 detector pixels 131 in a first direction, and the projection of microlens 1211 onto image sensor 130 covers a second number of 5 detector pixels 131 in a second direction, then in the optical axis direction, the projection of microlens 1211 onto image sensor 130 covers the area corresponding to 5×5 detector pixels 131. In this case, the preset number of detector pixels 131 covered by the projection of microlens 1211 onto image sensor 130 can be 25.

[0058] Of course, it is not limited to this. The first quantity and the second quantity can be the same or different. There are no restrictions here.

[0059] Based on the fact that the microlens 1211 included in the visual imaging module 100 of the image processing system 10 is projected onto the image sensor 130 in a first direction to cover a first number of detector pixels 131, and the microlens 1211 is projected onto the image sensor 130 in a second direction to cover a second number of detector pixels 131, the image processing system 10 can coordinate the microlens 1211 of the visual imaging module 100 and the detector pixels 131 covered by the microlens 1211 to improve the ease of acquiring multi-dimensional optical information of the light of the target object.

[0060] In one embodiment, the zoom lens group 110 includes a zoom device 111 and an imaging lens group 112; in the optical axis direction, the imaging lens group 112 is located between the zoom device 111 and the micro / nano optical device 120; the zoom device 111 is used to adjust the imaging focal length of the imaging lens group 112 to determine the imaging focal length of the zoom lens group 110.

[0061] For example, zoom device 111 can be used to adjust the imaging focal length of imaging lens group 112, thereby determining the imaging focal length of zoom lens group 110. For instance, zoom device 111 may include at least one of a tunable lens, a voice coil motor (VCM), and a zoom lens. The voice coil electrode can also be referred to as a VCM zoom motor. Of course, zoom device 111 is not limited to these, and no limitation is made here. In an exemplary embodiment, when zoom device 111 includes a tunable lens, the tunable lens can change its optical power in response to at least one received external signal, such as voltage, pressure, etc. When the optical power of the tunable lens changes, the combined focal length of the zoom lens group 110, including the tunable lens of the zoom device 111 and the imaging lens group 112, also changes accordingly. Therefore, the changed combined focal length of the zoom lens group 110, including the zoom device 111 and the imaging lens group 112, can be determined as the imaging focal length of the zoom lens group 110. In another exemplary embodiment, when the zoom device 111 includes a voice coil motor, the voice coil motor can be used to drive the imaging lens group 112 to move back and forth along the optical axis, thereby adjusting the imaging focal length of the imaging lens group 112. In response to the change in the imaging focal length of the imaging lens group 112, the combined focal length of the zoom lens group 110, including the voice coil motor of the zoom device 111 and the imaging lens group 112, also changes accordingly. Therefore, the changed combined focal length of the zoom lens group 110, including the zoom device 111 and the imaging lens group 112, can be determined as the imaging focal length of the zoom lens group 110. Of course, it is not limited to this, and no restrictions are set here.

[0062] For example, the imaging lens group 112 may include a third micro / nano optical device. This third micro / nano optical device may have imaging capabilities. For instance, the third micro / nano optical device may include a metasurface. However, the third micro / nano optical device is not limited to this, and no limitation is made here. When the imaging lens group 112 includes a third micro / nano optical device, because the third micro / nano optical device is small in size, the size of the zoom lens group 110 can be further reduced, thereby reducing the size of the visual imaging module 100, and consequently reducing the size of the image processing system 10.

[0063] Based on the fact that the imaging lens group 112 is located between the zoom device 111 and the micro / nano optical device 120 along the optical axis, it can be determined that the zoom device 111 is located in front of the imaging lens group 112 along the optical axis. The zoom device 111 can adjust the imaging focal length of the imaging lens group 112 to achieve a change in the back focal length of the visual imaging module 100, thereby determining the imaging focal length of the zoom lens group 110.

[0064] When the visual imaging module 100 can adjust the imaging focal length of the imaging lens group 112 using the zoom device 111 to determine the imaging focal length of the zoom lens group 110, the image processing system 10 can acquire light rays emitted or reflected from at least one target object in the target scene through the zoom lens group 110 of the visual imaging module 100. This allows the image processing system 10 to determine the multi-dimensional incident angle of the light rays from the target object, thereby improving the ease with which the image processing system 10 acquires multi-dimensional optical information of the light rays from the target object.

[0065] In one embodiment, the distance between the zoom lens group 110 and the micro / nano optical device 120 in the optical axis direction is greater than or equal to 13 mm and less than or equal to 120 mm.

[0066] For example, the distance between the zoom lens group 110 and the micro / nano optical device 120 along the optical axis can be designed based on the imaging focal length of the zoom lens group 110 itself. For instance, if the initial imaging focal length of the zoom lens group 110 is greater than or equal to 13mm and less than or equal to 120mm, the distance between the zoom lens group 110 and the micro / nano optical device 120 along the optical axis can be greater than or equal to 13mm and less than or equal to 120mm. Of course, the distance between the zoom lens group 110 and the micro / nano optical device 120 is not limited to this, and is not restricted here.

[0067] For example, the distance between the zoom lens group 110 and the micro / nano optical device 120 along the optical axis can be one of 13mm, 20mm, 25mm, 36mm, 47mm, 58mm, 64mm, 70mm, 88mm, 92mm, 106mm, 111mm, or 120mm. Of course, the distance between the zoom lens group 110 and the micro / nano optical device 120 is not limited to this, and is not restricted here.

[0068] Based on the fact that the distance between the zoom lens group 110 and the micro-nano optical device 120 in the optical axis direction is greater than or equal to 13 mm and less than or equal to 120 mm, the distance between the zoom lens group 110 and the micro-nano optical device 120 can be designed according to the optical path state output by the zoom lens group 110. This ensures that the light rays of the target object collected by the visual imaging module 100 are incident on the micro-nano optical device 120 at the correct angle and distribution. As a result, the image processing system 10 can adjust and / or collect the optical information of the light rays of the target object through the micro-nano optical device 120 of the visual imaging module 100, thereby improving the convenience of the image processing system 10 in acquiring multi-dimensional optical information of the target object.

[0069] In one embodiment, the focal plane of the zoom lens group 110 converges onto the micro / nano optical device 120.

[0070] As shown in Figure 6, when the focal plane of the zoom lens group 110 converges on the micro-nano optical device 120, it can be ensured that the light from the target object collected by the visual imaging module 100 can be transmitted to the micro-nano optical device 120 after passing through the zoom lens group 110, so that the micro-nano optical device 120 can adjust and / or collect the optical information of the light from the target object, thereby improving the convenience of the image processing system 10 in collecting multi-dimensional optical information of the light from the target object.

[0071] For a target object at the same object distance, the visual imaging module 100 can change the imaging focal length of the zoom lens group 110 to make the target object image in front of the micro-nano optical device 120, as shown in Figure 7a. The visual imaging module 100 can also change the imaging focal length of the zoom lens group 110 to make the target object image behind the micro-nano optical device 120, as shown in Figure 7b.

[0072] Accordingly, when the zoom lens group 110 is used to change the imaging focal length of the zoom lens group 110, for target objects at different object distances, the image processing system 10 can adjust the imaging focal length of the zoom lens group 110 through the zoom lens group 110 of the vision imaging module 100, so that the light rays from target objects at different object distances acquired by the vision imaging module 100 can be converged onto the micro / nano optical device 120 after being acted upon by the zoom lens group 110. For example, the vision imaging module 100 can acquire the object distance of the target object. When the vision imaging module 100 acquires the object distance of the target object, the imaging focal length required by the zoom lens group 110 when the vision imaging module 100 acquires a clear image of the target object can be determined based on the object distance. When the current imaging focal length of the zoom lens group 110 is inconsistent with the imaging focal length required by the visual imaging module 100 to acquire a clear image of the target object, the imaging focal length of the zoom lens group 110 can be changed according to the imaging focal length required by the visual imaging module 100 to acquire a clear image of the target object. When the current imaging focal length of the zoom lens group 110 is consistent with the imaging focal length required by the visual imaging module 100 to acquire a clear image of the target object, it can be determined that there is no need to change the imaging focal length of the zoom lens group 110. Based on this, the light emitted or reflected by the target object can be converged onto the micro / nano optical device 120 under the action of the zoom lens group 110 with or without the changed imaging focal length. Based on this, the light rays from target objects at different object distances that converge on the micro-nano optical device 120 can be used by the micro-nano optical device 120 to adjust and / or collect the optical information of the light rays from target objects at different object distances, thereby improving the ease with which the image processing system 10 can acquire multi-dimensional optical information of the light rays from the target objects.

[0073] Please refer to Figure 8, which is a schematic flowchart of an image processing method provided in an embodiment of this application. The image processing method is applied to an image processing system. The image processing system includes a visual imaging module and a data processing module; the visual imaging module is connected to the data processing module; the visual imaging module includes a zoom lens group, micro / nano optical devices, and an image sensor that are sequentially adjacent in the optical axis direction. For a more detailed description of the image processing system, please refer to the description of the image processing system in the foregoing embodiments, which will not be repeated here.

[0074] As shown in Figure 8, the image processing method includes steps S101 to S102.

[0075] S101. When the light emitted or reflected by the target object passes through the visual imaging module, an image of the target object is generated.

[0076] For example, when light emitted or reflected from a target object illuminates the visual imaging module in the image processing system, the visual imaging module can collect the light illuminating it and perform imaging processing on the target object based on the collected light to obtain an image of the target object.

[0077] In some implementations, the light emitted or reflected by the target object is transmitted to the image sensor included in the visual imaging module after being processed by the zoom lens group and micro-nano optical devices, thereby obtaining an image of the target object.

[0078] When the visual imaging module determines the image of the target object, the image of the target object can be used by the image processing system to subsequently determine the optical information of the target object's light rays, which is beneficial to improving the ease of determining the optical information of the target object's light rays by the image processing system.

[0079] S102. The data processing module performs data analysis on the image of the target object to obtain the optical information of the light rays of the target object; the optical information includes at least two of the following: light intensity, spectrum, polarization, and direction.

[0080] For example, when an image processing system determines an image of a target object through a visual imaging module, it can transmit the image of the target object to a data processing module. The data processing module then performs data analysis on the image of the target object to obtain the optical information of the target object's light rays. The optical information of the target object's light rays includes at least two of the following: light intensity, spectrum, polarization, and direction. In an exemplary embodiment, the image processing system can transmit the image of the target object determined by the visual imaging module to the data processing module based on the communication connection between the visual imaging module and the data processing module, so that the data processing module can determine the optical information of the target object's light rays. However, this is not a limitation and is not intended to restrict the scope of the system.

[0081] As shown in Figure 4, the image of the target object determined by the visual imaging module can be analyzed by the data processing module to obtain multi-dimensional optical information of the light emitted or reflected by the target object. For example, the data processing module can process the image of the target object provided by the visual imaging module based on at least one of the following methods: Neural Processing Unit (NPU), Application-Specific Integrated Circuit (ASIC), etc., to obtain the optical information of the light emitted by the target object. The optical information of the light emitted by the target object can include at least two of the following: light intensity, spectrum, polarization, and direction, without limitation. The data processing module can have at least one of the following data processing functions: numerical analysis function, deep learning function, etc., so that the data processing module can simultaneously analyze and determine the multi-dimensional optical information of the light emitted by the target object by performing data analysis on the image of the target object determined by the visual imaging module. For example, the data processing module can include a multimodal large model. The multimodal large model can be trained based on the images of different preset objects determined by the visual imaging module. Different preset objects can include the target object and other objects besides the target object. Of course, it is not limited to this, and no limitation is made here.

[0082] In some implementations, the image of the target object is preprocessed by the image signal processor of the data processing module to obtain a preprocessed image; the preprocessed image is then analyzed by at least one of the neural network processor and application-specific integrated circuit of the data processing module to obtain the optical information of the light source of the target object.

[0083] For example, the data processing module may include at least one of an image signal processor, a neural network processor, and a dedicated application-specific integrated network (ASIC). The data processing module can use the image signal processor to preprocess the image of the target object to obtain a preprocessed image. Preprocessing may include at least one of noise reduction, color difference, white balance, gamma correction, etc., of the image of the target object. After preprocessing the image of the target object using the image signal processor, the data processing module can use at least one of the neural network processor and ASIC to perform data analysis on the preprocessed image to obtain the optical information of the target object's light.

[0084] For example, multi-dimensional optical information can include at least two of the following: polarization, spectrum, intensity, and direction of light. For instance, an image processing system can use a data processing module combined with an image of a target object determined by a visual imaging module to simultaneously analyze and determine at least two of the optical information of the target object, such as polarization, spectrum, intensity, and direction of light. Of course, multi-dimensional optical information is not limited to this; for example, the data processing module can also determine the depth of field corresponding to the target object based on its image, without further limitation.

[0085] When an image processing system determines the multi-dimensional optical information of the light rays of a target object, such as at least two of the polarization, spectrum, intensity, and direction of the light rays, the multi-dimensional optical information of the light rays of the target object can be used for multimodal object detection, removal of strong reflections in photography, acquisition of multi-dimensional incident angles, etc., which helps to improve the image processing system's ability to perceive the real target scene in which the target object is located.

[0086] When the image of the target object determined by the visual imaging module can be analyzed by the data processing module of the image processing system to obtain the optical information of the target object's light rays, such as at least two of the optical information of the target object's light rays, including polarization, spectrum, light intensity, and direction, it is beneficial to improve the ease of determining the optical information of the target object's light rays by the image processing system.

[0087] In some implementations, when the incident angle of the light emitted or reflected by the target object onto the visual imaging module is within the incident angle range corresponding to the visual imaging module, the visual imaging module performs imaging processing on the light emitted or reflected by the target object to obtain an image of the target object.

[0088] For example, the visual imaging module has a corresponding viewing angle. The viewing angle of the visual imaging module can change in response to changes in the imaging focal length of the zoom lens group. The viewing angle of the visual imaging module can be used to determine the range of incident angles of light that the visual imaging module can collect and use for imaging; therefore, the viewing angle of the visual imaging module can also be called the incident angle range corresponding to the visual imaging module. When the incident angle of light emitted or reflected from the target object is within the incident angle range corresponding to the visual imaging module, the visual imaging module can collect the light emitted or reflected from the target object and perform imaging processing on the light emitted from the target object to determine an image of the target object.

[0089] The image of the target object can be used by the image processing system to subsequently determine the optical information of the light rays emanating from the target object, such as the direction of the light rays. For example, the image processing system can use the data processing module to analyze the image of the target object determined by the visual imaging module, obtain the incident angle of the light rays from the target object when they hit the visual imaging module, and then determine the direction of the light rays from the target object based on the incident angle. This will help improve the ease of determining the direction of the light rays from the target object in subsequent image processing systems.

[0090] In some implementations, if the object distance of the target object is outside the depth of field of the visual imaging module, the imaging focal length of the zoom lens group is changed so that the object distance of the target object is within the depth of field of the visual imaging module; when the object distance of the target object is within the depth of field of the visual imaging module, and the light emitted or reflected by the target object passes through the visual imaging module, an image of the target object is generated.

[0091] For example, when the target scene includes multiple target objects, the object distances of the different target objects can be different. The visual imaging module can have a specific depth of field range. If the object distance of the target object is within the depth of field range of the visual imaging module, the visual imaging module can acquire a clear image of the target object. If the object distance of the target object is outside the depth of field range of the visual imaging module, the visual imaging module can only acquire a blurry image of the target object. When the visual imaging module includes a zoom lens group, the visual imaging module can change the depth of field range of the visual imaging module by changing the imaging focal length of the zoom lens group, thereby ensuring that target objects at different object distances are all within the depth of field range of the visual imaging module, thus acquiring clear images of the target objects. For example, when the focal length of the zoom lens group changes, the visual imaging module can focus in conjunction with itself to dynamically adjust its depth of field. This ensures that targets at different distances fall within the depth of field of the visual imaging module, meaning they are all within its sharp imaging range. When light emitted or reflected from the target object passes through the visual imaging module, it processes the light to generate an image of the target object. When the target object's distance is within the depth of field of the visual imaging module, the resulting image is sharp. This sharp image can then be analyzed by the image processing system's data processing module to determine the optical information of the target object's light rays, such as their direction. However, this is not a limitation and is not set forth here.

[0092] For example, the micro-nano optical device includes a first micro-nano optical device and a second micro-nano optical device. The first micro-nano optical device and the second micro-nano optical device can enable an image processing system to acquire and / or adjust at least one of the polarization, spectrum, and intensity of light from a target object.

[0093] In some implementations, light emitted or reflected from the target object is transmitted to an image sensor after passing through a first micro-nano optical device and a second micro-nano optical device to generate an image of the target object.

[0094] For example, light emitted or reflected from the target object is transmitted to an image sensor after passing through a zoom lens group, a first micro-nano optical device, and a second micro-nano optical device, in order to obtain an image of the target object.

[0095] For example, when the first micro-nano optical device is located between the zoom lens group and the second micro-nano optical device in the optical axis direction, the light emitted or reflected by the target object can be transmitted to the image sensor after passing through the zoom lens group, the first micro-nano optical device and the second micro-nano optical device in sequence, so as to obtain an image of the target object.

[0096] For example, when the second micro-nano optical device is located between the zoom lens group and the second micro-nano optical device in the optical axis direction, the light emitted or reflected by the target object can be transmitted to the image sensor after passing through the zoom lens group, the second micro-nano optical device and the first micro-nano optical device in sequence, so as to obtain an image of the target object.

[0097] When the micro-nano optical device includes a first micro-nano optical device and a second micro-nano optical device, the image processing system can simultaneously acquire and / or adjust the multi-dimensional optical information of the light from the target object using both the first and second micro-nano optical devices of the visual imaging module. For example, the first micro-nano optical device can be used to acquire and / or adjust the polarization of the light from the target object, and the second micro-nano optical device can be used to acquire and / or adjust the spectrum of the light from the target object. However, this is not a limitation; the image processing system can also simultaneously acquire and / or adjust the multi-dimensional optical information of the light from the target object using either the first or second micro-nano optical device of the visual imaging module. Thus, when the micro-nano optical device includes both a first and a second micro-nano optical device, it improves the ease of acquisition and adjustment of the multi-dimensional optical information of the light from the target object by the image processing system.

[0098] In some embodiments, light emitted or reflected from the target object is transmitted to an image sensor after passing through a zoom lens group, at least one microlens included in the first micro-nano optical device, and a second micro-nano optical device, in order to obtain an image of the target object.

[0099] For example, the first micro-nano optical device may include multiple microlenses. Each microlens included in the first micro-nano optical device may serve as a macropixel unit for acquiring and / or adjusting optical information of light from a target object.

[0100] For example, light emitted or reflected from the target object can be transmitted to an image sensor after passing through a zoom lens group, at least one microlens included in the first micro-nano optical device, and a second micro-nano optical device in sequence, so as to obtain an image of the target object.

[0101] Based on this, in the process of determining the image of the target object through the visual imaging module, the image processing system can coordinate at least one microlens of the first micro-nano optical device and the second micro-nano optical device to collect and / or adjust the optical information of the light of the target object. This will help improve the ease of collecting and adjusting the multi-dimensional optical information of the light of the target object by the image processing system.

[0102] In some embodiments, the micro-nano optical device includes multiple microlenses. Along the optical axis, the projection of each microlens onto the image sensor covers a predetermined number of detection units included in the image sensor. When light emitted or reflected from the target object passes through the microlenses of the visual imaging module, the microlenses focus the light emitted or reflected from the target object at different incident angles, so that the light at different incident angles converges onto the corresponding detection pixels to generate an image of the target object. The image includes optical information of light at different incident angles.

[0103] As shown in Figure 5, the projection of each microlens onto the image sensor along the optical axis can be considered as the equivalent region of the corresponding microlens on the image sensor. The equivalent region of each microlens is used to cover a predetermined number of detector pixels included in the image sensor. This predetermined number includes at least two.

[0104] When the projection of the microlens onto the image sensor along the optical axis covers a preset number of detector pixels included in the image sensor, the light processed by the microlens is transmitted to the detector pixels corresponding to the microlens on the image sensor, thereby enabling the visual imaging module to determine the image of the target object.

[0105] In one exemplary embodiment, the projection of the microlens onto the image sensor is directed upwards in a first direction to cover a first number of detector pixels; the projection of the microlens onto the image sensor is directed in a second direction to cover a second number of detector pixels; wherein the first direction is perpendicular to the second direction, and both the first and second directions are perpendicular to the optical axis. The image processing system can determine that a preset number of detector pixels corresponding to the microlens on the image sensor is equal to the product of the first number and the second number. Therefore, the image processing system can coordinate with the preset number of detector pixels to determine the image of the target object, which improves the ease of image determination of the target object.

[0106] Furthermore, the visual imaging module can work in conjunction with microlenses and the detector pixels covered by the microlenses to acquire optical information about the light emitted from the target object. For example, the image processing system can use the microlenses of the visual imaging module to focus light emitted or reflected from the target object at different incident angles, converging the light at different incident angles onto different detector pixels covered by the microlenses. This allows each detector pixel to acquire optical information about the light at different incident angles, such as the direction of the light at different incident angles. Based on this, the image processing system can work in conjunction with the microlenses of the visual imaging module and the detector pixels covered by the microlenses to improve the ease of acquiring multi-dimensional optical information about the light emitted from the target object.

[0107] In some embodiments, light emitted or reflected from the target object is transmitted to an image sensor after passing through the zoom device included in the zoom lens group, the imaging lens group included in the zoom lens group, and the micro-nano optical devices, so as to obtain an image of the target object.

[0108] For example, a zoom lens group includes a zooming device and an imaging lens group. The zooming device is used to adjust the imaging focal length of the imaging lens group to determine the imaging focal length of the zoom lens group. For instance, the zooming device may include at least one of a tunable lens, a voice coil motor (VCM), and a zoom lens. The voice coil electrode can also be referred to as a VCM zoom motor. Of course, the zooming device is not limited to these, and no limitation is made here.

[0109] For example, along the optical axis, the imaging lens group is located between the zoom device and the micro / nano optical device. Light emitted or reflected from the target object can be imaged onto the micro / nano optical device after passing through the zoom device and the imaging lens group in sequence, and then transmitted to the image sensor after passing through the micro / nano optical device to obtain an image of the target object.

[0110] Based on this, in the process of determining the image of the target object through the visual imaging module, the image processing system can coordinate with the zoom lens group, including the zoom device and the imaging lens group, so that the light from the target object is imaged onto the micro-nano optical device, which can then be used by the image sensor to determine the image of the target object. This helps to improve the convenience and accuracy of the image processing system in determining the image of the target object.

[0111] In some implementations, light emitted or reflected from the target object is transmitted to an image sensor after passing through a zoom device, a third micro / nano optical device included in the imaging lens group, and other micro / nano optical devices, in order to obtain an image of the target object.

[0112] For example, the imaging lens group includes a third micro / nano optical device. This third micro / nano optical device can have imaging capabilities. For instance, the third micro / nano optical device includes a metasurface. Of course, the third micro / nano optical device is not limited to this, and no limitation is made here. When the imaging lens group includes a third micro / nano optical device, because the third micro / nano optical device is small in size, the size of the zoom lens group can be further reduced, thereby reducing the size of the visual imaging module and, consequently, the size of the image processing system.

[0113] Accordingly, since the imaging lens group is located between the zoom device and the micro / nano optical device along the optical axis, when the imaging lens group includes a third micro / nano optical device, the third micro / nano optical device is located between the zoom device and the micro / nano optical device. Light emitted or reflected from the target object, after being processed by the zoom device and the third micro / nano optical device, can be imaged onto the micro / nano optical device, and then transmitted to the image sensor after being processed by the micro / nano optical device to obtain an image of the target object.

[0114] Based on this, in the process of determining the image of the target object through the visual imaging module, the image processing system can coordinate with the zoom device included in the zoom lens group and the third micro-nano optical device included in the imaging lens group, so that the light from the target object is imaged onto the micro-nano optical device, which can then be used by the image sensor to determine the image of the target object. This helps to improve the convenience and accuracy of the image processing system in determining the image of the target object.

[0115] In some implementations, light rays passing through a zoom lens group are focused onto a micro / nano optical device.

[0116] For example, if the focal plane of the zoom lens group converges on the micro-nano optical device, then the light emitted or reflected by the target object can be transmitted to the micro-nano optical device after being processed by the zoom lens group, and then transmitted to the image sensor after being processed by the micro-nano optical device, so that the image sensor can determine the image of the target object, thereby improving the convenience and accuracy of the image processing system in determining the image of the target object.

[0117] In some implementations, before the data processing module performs data analysis on the image of the target object to obtain the optical information of the target object's light rays, the image processing method includes: acquiring a training sample set, which includes images of multiple preset objects determined by the visual imaging module, and optical information tags corresponding to the images of each preset object; performing data analysis on the images of the preset objects based on the data processing module to obtain the optical information of the preset objects' light rays; determining optimization parameters for the data processing module based on the optical information tags and the optical information of the preset objects' light rays; and optimizing the data processing module based on the optimization parameters to obtain an optimized data processing module.

[0118] For example, before the data processing module performs data analysis on the image of the target object to obtain the optical information of the target object's light, the data processing module can be trained to enable it to perform data analysis on the image of the target object and obtain the optical information of the target object's light.

[0119] For example, when training a data processing module, a training sample set can be obtained. The training sample set may include images of multiple preset objects. These preset objects may include the target object as well as other objects besides the target object, to enhance the diversity of the training sample set. The images of the preset objects are acquired or generated by the visual imaging module in the image processing system. Each image of a preset object may be assigned a corresponding optical information label. The optical information label may include at least two of the following: light intensity label, spectral label, polarization label, and direction label of the target object's light rays. The optical information label may be pre-labeled and is not limited here.

[0120] With a training sample set available, images of a preset object included in the training sample set can be input into a data processing module. The module then analyzes these images to obtain the optical information of the object's rays. This optical information, determined by the data processing module, can be used to evaluate its consistency with the optical information label corresponding to the object's image. For example, the optical information determined by the data processing module is equivalent to the predicted optical information of the object's rays. The optical information label corresponding to the object's image is equivalent to the actual optical information of the object's rays. Based on the optical information and the optical information label, the loss value between the predicted and actual optical information when determining the object's rays can be determined. A larger loss value indicates a greater difference between the predicted and actual optical information. Conversely, a smaller loss value indicates a smaller difference between the predicted and actual optical information. Based on this, the data processing module can determine its optimization parameters according to the loss value, and then optimize the data processing module according to the optimization parameters. This allows the optimized data processing module to more accurately align the predicted optical information of the light rays of the preset object with the actual optical information of the preset object, thereby achieving the training objective. The optimized data processing module can then be used as a data processing module in the image processing system, allowing the image processing system to perform data analysis on the image of the target object determined by the visual imaging module to obtain the optical information of the light rays of the target object. The optical information includes at least two of the following: light intensity, spectrum, polarization, and direction. However, this is not limited to this. During the process of determining the optical information of the light rays of the target object using the data processing module, the image processing system can continuously optimize the data processing module using the optical information of the light rays determined by the data processing module and the corresponding optical information tags of the target object image. This is not a limitation.

[0121] By training the data processing module using a training sample set to optimize it, the optimized data processing module can be used as a data processing module included in an image processing system. Furthermore, based on the image of the target object determined by the visual imaging module, the optical information of the target object's light can be determined, which helps to improve the ease with which the image processing system can acquire multi-dimensional optical information of the target object's light.

[0122] Please refer to Figure 9, which is a schematic block diagram of an image processing apparatus provided in an embodiment of this application.

[0123] As shown in Figure 9, this image processing device can be configured in an image processing system to execute the aforementioned image processing method. The image processing system includes a visual imaging module and a data processing module; the visual imaging module is connected to the data processing module; the visual imaging module includes a zoom lens group, micro / nano optical devices, and an image sensor that are sequentially adjacent in the optical axis direction.

[0124] As shown in Figure 9, the image processing device includes an image acquisition module 310 and a data analysis module 320.

[0125] The image acquisition module 310 is used to generate an image of the target object when the light emitted or reflected by the target object passes through the visual imaging module of the image processing system.

[0126] The data analysis module 320 is used to perform data analysis on the image of the target object through the data processing module of the image processing system to obtain the optical information of the light of the target object; the optical information includes at least two of the following: light intensity, spectrum, polarization, and direction.

[0127] For example, the image acquisition module 310 includes a first acquisition submodule.

[0128] The first acquisition submodule is used to obtain an image of the target object by performing imaging processing on the light emitted or reflected by the target object based on the visual imaging module when the light emitted or reflected by the target object is incident on the incident angle range corresponding to the visual imaging module.

[0129] For example, the image acquisition module 310 includes a zoom submodule and a second acquisition submodule.

[0130] The zoom submodule is used to change the imaging focal length of the zoom lens group so that the object distance of the target object is within the depth of field of the visual imaging module if the object distance of the target object is outside the depth of field of the visual imaging module.

[0131] The second acquisition submodule is used to generate an image of the target object when the object distance of the target object is within the depth of field of the visual imaging module, and when the light emitted or reflected by the target object passes through the visual imaging module.

[0132] For example, the micro-nano optical device includes multiple microlenses, and in the optical axis direction, the projection of each microlens onto the image sensor is used to cover a preset number of detection units included in the image sensor; the image acquisition module 310 includes a third acquisition submodule.

[0133] The third acquisition submodule is used to focus the light emitted or reflected by the target object at different incident angles through the microlens of the visual imaging module when the light emitted or reflected by the target object passes through the microlens, so that the light at different incident angles converges onto the corresponding detection pixel to generate an image of the target object, wherein the image includes optical information of light at different incident angles.

[0134] For example, the image processing device includes a training sample acquisition submodule, an optical information determination submodule, an optimization parameter determination submodule, and a module optimization submodule.

[0135] The training sample acquisition submodule is used to acquire a training sample set, which includes images of multiple preset objects determined by the visual imaging module, and optical information tags corresponding to the images of each preset object.

[0136] The optical information determination submodule is used to perform data analysis on the image of the preset object based on the data processing module to obtain the optical information of the light rays of the preset object.

[0137] The optimization parameter determination submodule is used to determine the optimization parameters of the data processing module based on the optical information of the optical information tag and the light of the preset object.

[0138] The module optimization submodule is used to optimize the data processing module according to the optimization parameters to obtain an optimized data processing module.

[0139] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and its modules and units can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0140] The method of this application can be used in image processing systems, which can be used in a variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0141] For example, the above-described method and apparatus can be implemented as a computer program that runs on an image processing system or server to control the image processing system. For example, the image processing system can be applied to electronic devices such as AR glasses, VR glasses, MR glasses, AR headsets, VR headsets, and MR headsets, without limitation. The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0142] Please refer to Figure 10, which is a schematic block diagram of an image processing system provided in an embodiment of this application. As shown in Figure 10, the image processing system includes a visual imaging module and a data processing module; the visual imaging module is connected to the data processing module. The visual imaging module includes a zoom lens group, micro / nano optical devices, and an image sensor (not shown in the figure) that are sequentially adjacent in the optical axis direction. The image processing system includes a memory and a processor. The memory and the processor can be connected via a system bus, and the memory may include a storage medium and internal memory.

[0143] The storage medium can store the operating system and computer programs. When a computer program is executed, it can cause the processor to perform any image processing method.

[0144] The processor provides computing and control capabilities to support the operation of the entire image processing system.

[0145] Internal memory provides an environment for the execution of computer programs stored in the storage medium. When these computer programs are executed by the processor, the processor can perform any image processing method.

[0146] Those skilled in the art will understand that the structure shown in Figure 10 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the image processing system to which the present application is applied. A specific image processing system may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0147] It should be understood that a processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other convertible logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0148] In one embodiment, the processor is configured to execute a computer program and, when executing the computer program, perform the following steps: when light emitted or reflected from a target object passes through the visual imaging module, an image of the target object is generated; the image of the target object is analyzed by the data processing module to obtain optical information of the light emitted by the target object; the optical information includes at least two of the following: light intensity, spectrum, polarization, and direction.

[0149] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of image processing described above can be referred to the corresponding process in the foregoing image processing system embodiments and / or image processing method embodiments, and will not be repeated here.

[0150] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method implemented can be referred to in various embodiments of the image processing method of this application.

[0151] The computer-readable storage medium can be an internal storage unit of the image processing system described in the foregoing embodiments, such as the hard disk or memory of the image processing system. Alternatively, the computer-readable storage medium can be an external storage device of the image processing system, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the image processing system.

[0152] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0153] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0154] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above descriptions are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, This is applied to an image processing system, which includes a visual imaging module and a data processing module; the visual imaging module is connected to the data processing module. The visual imaging module includes a zoom lens group, micro-nano optical devices, and an image sensor that are sequentially adjacent in the optical axis direction. The image processing method includes: generating an image of the target object when light emitted or reflected from the target object passes through the visual imaging module; The data processing module performs data analysis on the image of the target object to obtain the optical information of the light rays of the target object; the optical information includes at least two of the following: light intensity, spectrum, polarization, and direction.

2. The image processing method according to claim 1, characterized in that, The step of generating an image of the target object when light emitted or reflected from the target object passes through the visual imaging module includes: when the incident angle of light emitted or reflected from the target object onto the visual imaging module is within the incident angle range corresponding to the visual imaging module, performing imaging processing on the light emitted or reflected from the target object based on the visual imaging module to obtain an image of the target object.

3. The image processing method according to claim 1, characterized in that, The step of generating an image of the target object when light emitted or reflected from the target object passes through the visual imaging module includes: if the object distance of the target object is outside the depth of field of the visual imaging module, changing the imaging focal length of the zoom lens group so that the object distance of the target object is within the depth of field of the visual imaging module; and generating an image of the target object when the object distance of the target object is within the depth of field of the visual imaging module and light emitted or reflected from the target object passes through the visual imaging module.

4. The image processing method according to claim 1, characterized in that, The micro-nano optical device includes a first micro-nano optical device and a second micro-nano optical device; The step of generating an image of the target object when light emitted or reflected from the target object passes through the visual imaging module includes: the light emitted or reflected from the target object is transmitted to the image sensor after being processed by the first and second micro-nano optical devices included in the micro-nano optical device, so as to generate an image of the target object.

5. The image processing method according to claim 1, characterized in that, The micro-nano optical device includes multiple microlenses, and in the optical axis direction, the projection of each microlens onto the image sensor is used to cover a predetermined number of detection units included in the image sensor; The step of generating an image of the target object when light emitted or reflected from the target object passes through the visual imaging module includes: when light emitted or reflected from the target object passes through the microlens of the visual imaging module, focusing light emitted or reflected from the target object at different incident angles through the microlens, so that light at different incident angles converges onto the corresponding detection pixel to generate an image of the target object, wherein the image includes optical information of light at different incident angles.

6. The image processing method according to any one of claims 1 to 5, characterized in that, The step of performing data analysis on the image of the target object through the data processing module to obtain the optical information of the light rays of the target object includes: preprocessing the image of the target object through the data processing module to obtain a preprocessed image; and performing data analysis on the preprocessed image through at least one of the neural network processor and dedicated integrated circuit of the data processing module to obtain the optical information of the light rays of the target object.

7. The image processing method according to any one of claims 1 to 5, characterized in that, Before performing data analysis on the image of the target object using the data processing module to obtain the optical information of the target object's light rays, the image processing method includes: acquiring a training sample set, the training sample set including images of multiple preset objects determined by the visual imaging module, and optical information tags corresponding to the images of each preset object; performing data analysis on the images of the preset objects based on the data processing module to obtain the optical information of the preset objects' light rays; determining optimization parameters for the data processing module based on the optical information tags and the optical information of the preset objects' light rays; and optimizing the data processing module based on the optimization parameters to obtain an optimized data processing module.

8. An image processing apparatus, characterized in that, The image processing device includes: an image acquisition module, used to generate an image of the target object when light emitted or reflected from the target object passes through the visual imaging module of the image processing system; and a data analysis module, used to perform data analysis on the image of the target object through the data processing module of the image processing system to obtain optical information of the light emitted from the target object; the optical information includes at least two of the following: light intensity, spectrum, polarization, and direction.

9. An image processing system, characterized in that, The image processing system includes a visual imaging module and a data processing module; the visual imaging module is connected to the data processing module. The visual imaging module includes a zoom lens group, micro / nano optical devices, and an image sensor arranged sequentially along the optical axis. The zoom lens group is used to change the imaging focal length of the zoom lens group. The micro / nano optical devices are used to collect and / or adjust the optical information of light. Light emitted or reflected from the target object is transmitted to the image sensor after passing through the zoom lens group and the micro / nano optical devices to obtain an image of the target object. The data processing module is used to acquire the image of the target object determined by the visual imaging module and to perform data analysis on the image of the target object to obtain the optical information of the light emitted by the target object. The optical information includes at least two of the following: light intensity, spectrum, polarization, and direction of the light emitted from the target object.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the steps of the image processing method as described in any one of claims 1 to 7.

Images