Visual imaging method and system of robot, robot, medium and product

Abstract

The invention discloses a visual imaging method and system of a robot, the robot, a medium and a product. The method comprises the following steps: performing polarization spectrum modulation on incident light through a modulation module to obtain multi-wavelength multi-polarization state incident light; the polarization spectrum modulation comprises time-sharing spectrum modulation and space-domain polarization modulation; irradiating the incident light to the surface of a to-be-measured target, and collecting multi-wavelength multi-polarization-state emergent light reflected by the surface of the to-be-measured target through an imaging module; determining an incident light Stokes vector corresponding to the incident light in the multi-wavelength and multi-polarization state, and performing demodulation calculation on the emergent light to obtain an emergent light Stokes vector; and inputting the incident light stokes vector and the emergent light stokes vector into a light transmission model to solve a muller matrix of a to-be-detected target, and decomposing a muller parameter image set from the muller matrix of the to-be-detected target. The information dimension of visual imaging is expanded, and the quality of visual imaging is improved.

Description

Technical Field

[0001] This invention relates to the field of visual imaging technology, and in particular to a visual imaging method, system, robot, medium, and product for a robot. Background Technology

[0002] Robot vision systems are a crucial component in robot inspection processes, enabling rapid and accurate detection of data anomalies through image recognition and analysis. However, with continuous improvements in algorithms and computing power, data-driven vision capabilities are gradually reaching a bottleneck. From the perspective of practical application, the accuracy of current quality inspection results often depends more on the quality of the raw data collected.

[0003] Existing robot vision systems use a single linear / area array camera to directly acquire the light intensity information of the target. Although this imaging system is simple in structure and low in cost, the acquired light intensity information of the target needs improvement, mainly in terms of quantity and quality. In terms of quantity, the image data acquired per unit time has few dimensions and a single pattern; in terms of quality, the acquired image data has high noise and weak target features. Summary of the Invention

[0004] This invention provides a visual imaging method, system, robot, medium, and product for robots, to solve the problem that existing robot vision systems use a single linear / area array camera to directly collect the light intensity information of the target, resulting in low quality and limited dimensionality of the collected target data.

[0005] In a first aspect, embodiments of the present invention provide a visual imaging method for a robot, comprising:

[0006] The incident light is polarized by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectral modulation includes time-division spectral modulation and spatial polarization modulation.

[0007] The incident light is irradiated onto the surface of the target under test, and the outgoing light with multiple wavelengths and multiple polarization states reflected from the surface of the target under test is collected by the imaging module.

[0008] Determine the incident light Stokes vector corresponding to the incident light in the multi-wavelength and multi-polarization states, and demodulate the outgoing light to obtain the outgoing light Stokes vector;

[0009] The incident light Stokes vector and the outgoing light Stokes vector are input into the optical transmission model to solve for the Müller matrix of the target under test, and the Müller parameter image set is decomposed from the Müller matrix of the target under test.

[0010] Secondly, embodiments of the present invention provide a robot vision system, comprising:

[0011] Modulation module, polarization-independent beam splitter, imaging module, and host computer;

[0012] The modulation module is used to receive control from the host computer, convert the incident light into incident light with multiple wavelengths and multiple polarization states through a multi-wavelength filter and a focal plane polarizer, and then irradiate the surface of the target under test with the incident light.

[0013] The polarization-independent beam splitter is used to separate the multi-wavelength, multi-polarization outgoing light reflected from the surface of the target under test from the main optical path to the imaging module.

[0014] The imaging module is used to collect outgoing light with multiple wavelengths and multiple polarization states reflected from the surface of the target under test;

[0015] The host computer is used to determine the incident light Stokes vector corresponding to the incident light in the multi-wavelength and multi-polarization state, and to demodulate the outgoing light to obtain the outgoing light Stokes vector; the incident light Stokes vector and the outgoing light Stokes vector are input into the optical transmission model to solve the Müller matrix of the target under test, and the Müller parameter image set is decomposed from the Müller matrix of the target under test.

[0016] Thirdly, embodiments of the present invention provide a robot, the robot comprising:

[0017] At least one processor;

[0018] and a memory communicatively connected to the at least one processor;

[0019] The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the visual imaging method of the robot according to any embodiment of the present invention.

[0020] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the visual imaging method of the robot according to any embodiment of the present invention.

[0021] Fifthly, embodiments of the present invention provide a computer program product including a computer program, which, when executed by a processor, implements the visual imaging method of a robot according to any embodiment of the present invention.

[0022] The technical solution of this invention involves obtaining incident light with multiple wavelengths and multiple polarization states by performing polarization spectrum modulation on the incident light through a modulation module; the polarization spectrum modulation includes time-division spectral modulation and spatial polarization modulation; the incident light is irradiated onto the surface of the target under test, and the outgoing light with multiple wavelengths and multiple polarization states reflected from the surface of the target under test is collected by an imaging module; the incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the outgoing light Stokes vector is obtained by demodulating the outgoing light; the incident light Stokes vector and the outgoing light Stokes vector are input into the optical transmission model to solve the Müller matrix of the target under test, and the Müller parameter image set is decomposed from the Müller matrix of the target under test. By employing time-division spectral modulation and spatial polarization modulation of incident light, the number of images acquired in a single camera acquisition action is increased. By decomposing the Müller matrix of the target under test, the acquired image information is expanded and enhanced, thus broadening the information dimension of visual imaging. Extending from the traditional single light intensity dimension to three information dimensions—light intensity, spectrum, and polarization—it reduces the impact of ambient light on the robot's visual capabilities, enhances the detection capability of small targets, and improves the quality of visual imaging. This solves the problem of existing robot vision systems using a single linear / area array camera to directly acquire the target's light intensity information, resulting in low-quality and single-dimensional target data. It has the beneficial effects of expanding the information dimension of visual imaging and improving visual imaging quality.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart of a visual imaging method for a robot provided in Embodiment 1 of the present invention;

[0026] Figure 2 A schematic diagram of a focal plane polarized pixel;

[0027] Figure 3 This is a schematic diagram of the structure of a robot vision system provided in Embodiment 2 of the present invention;

[0028] Figure 4 This is a schematic diagram of another robot vision system provided in Embodiment 2 of the present invention.

[0029] Figure 5 A schematic diagram of the robot structure for implementing the visual imaging method of the robot in this embodiment of the invention;

[0030] Figure 6 This is a schematic diagram of the robot's vision module. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Existing robot vision systems often use a single linear / area array camera to directly acquire the light intensity information of the target. Although this imaging system is simple in structure and low in cost, it also has the following disadvantages: Insufficient quantity: the image data acquired per unit time has few dimensions and a single pattern; Insufficient quality: the acquired image data has high noise and weak target features.

[0034] To address the issue of insufficient image data, the commonly used methods are data augmentation using generative size models (Generative Adversarial Networks, Variational Autoencoders, Diffusion Models, etc.) or simulation based on physical methods. However, neither of these methods can accurately reproduce the true physical information of the target.

[0035] To address the issue of insufficient image data quality, current solutions focus on image preprocessing (image denoising, contrast adjustment, grayscale co-occurrence, etc.). These methods can only subtract from the existing information of the target image, meaning that the generated image information is a proper subset of the source image information.

[0036] In response to the above-mentioned problems, this application combines multispectral imaging and active polarization imaging technologies to specially design and improve the traditional optical system and apply it to robot inspection vision scenarios.

[0037] Example 1

[0038] Figure 1 This is a flowchart of a robot visual imaging method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where visual imaging is achieved in robot inspection visual scenarios. The method can be executed by a robot vision system, which can be implemented in hardware and / or software and can be configured within the robot. Figure 1 As shown, the method includes:

[0039] S110. The incident light is obtained by polarization spectrum modulation through the modulation module to obtain incident light with multiple wavelengths and multiple polarization states; polarization spectrum modulation includes time-division spectrum modulation and spatial polarization modulation.

[0040] The robot vision system includes a modulation module and an imaging module. The modulation module modulates the polarization spectrum of the incident light. The imaging module forms an image of the target based on the outgoing light reflected from the surface of the target by the incident light. This disclosure does not limit the type of incident light; for example, a parallel beam can be used. The parallel beam can be formed by illuminating the target with a white light source and integrating it into a parallel beam through a lens group and an aperture.

[0041] Polarization spectral modulation can be considered as modulating incident light in two optical dimensions: spectrum and polarization. Polarization spectral modulation includes time-division spectral modulation and spatial polarization modulation. Time-division spectral modulation can be understood as modulating incident light in the time domain along the spectral dimension. Spatial polarization modulation can be understood as modulating incident light in the spatial domain along the polarization dimension. Multi-wavelength, multi-polarization incident light can include multiple incident lights of different wavelengths and polarization states.

[0042] The modulation module may include multi-wavelength filters and focal plane polarizers. Multi-wavelength filters can be a combination of filters with different fixed wavelengths, such as a 635 nm red light filter, a 532 nm green light filter, and a 488 nm blue light filter. A focal plane polarizer is a micro / nano optical device that miniaturizes and arrays the function of a traditional large polarizer into polarization analysis units distributed across the entire focal plane of the camera (i.e., the sensor target surface).

[0043] Specifically, the incident light is injected into the modulation module; the incident light is subjected to time-division spectral modulation through a multi-wavelength filter to obtain multi-wavelength incident light; and the incident light of each wavelength is subjected to spatial polarization modulation through a focal plane polarizer to obtain incident light with multiple wavelengths and multiple polarization states.

[0044] For example, in the modulation module, the host computer controls a rotary motor to drive the filter wheel to time-division multiplex the filters of each fixed wavelength in the multi-wavelength filter array to achieve time-division spectral modulation of the parallel beam of incident light, thus converting the incident light into multiple incident light of different wavelengths. Then, the incident light of each wavelength is further subjected to spatial polarization modulation by a focal plane polarizer, converting the incident light of each wavelength into multiple incident light of different polarization states, thereby obtaining multi-wavelength, multi-polarization incident light.

[0045] S120. The incident light is irradiated onto the surface of the target under test, and the outgoing light in multiple wavelengths and multiple polarization states reflected from the surface of the target under test is collected by the imaging module.

[0046] The target to be tested can be understood as the target that the robot needs to identify, such as an object to be tested. For example, in a data center inspection scenario, the target to be tested could be a data center cabinet.

[0047] The imaging module includes a lens group and a polarization camera. The lens group is an optical system composed of multiple lenses (optical lenses) arranged in a specific order and spacing to perform the imaging function. The polarization camera is a specialized camera capable of detecting and measuring the polarization state of light waves, converting invisible "polarization information" into visible images and data.

[0048] For example, in order to meet the detection stability of the multispectral and size of the target to be detected in the vision system, an achromatic telecentric lens group can be used in this embodiment. Thanks to the achromatic design, color aberrations can be avoided, and images with sharp edges and high contrast can be obtained. Furthermore, thanks to the telecentric design, small height fluctuations or positional changes of the object can be ignored.

[0049] Specifically, incident light is shone onto the surface of the target under test, and reflected off the surface of the target to form reflected light. The reflected light from the target is collected by the imaging module of the robot vision system. Since the incident light is in a multi-wavelength and multi-polarization state, the outgoing light collected by the imaging module is also in a multi-wavelength and multi-polarization state.

[0050] Optionally, the step of collecting the multi-wavelength, multi-polarization outgoing light reflected from the surface of the target under test by the imaging module includes: separating the outgoing light reflected from the surface of the target under test from the main optical path to the imaging module by a polarization-independent beam splitter; focusing the multi-wavelength, multi-polarization outgoing light by the lens group; and acquiring the focal plane polarization pixels of the outgoing light under a preset exposure by the polarization camera, wherein the focal plane polarization pixels are composed of four degrees of polarization.

[0051] Among them, a focal plane polarization pixel is an imaging pixel structure whose core function is to simultaneously acquire light intensity information from the same scene but with different polarization directions on the same focal plane (sensor surface) in a single exposure. A polarization-independent beamsplitter is a special type of optical beamsplitter whose splitting ratio (i.e., the ratio of transmitted to reflected light intensity) does not depend on the polarization state of the incident light.

[0052] Specifically, considering that the target under test may have a certain height, which could lead to changes in the incident angle in the vertical direction, this disclosure employs a polarization-independent beam splitter to separate the outgoing light reflected from the surface of the target under test from the main optical path and transmit it to the imaging module. In the imaging module, a DoFP polarization camera can be used to collect outgoing photons with different polarization states in real time. The polarization camera has several focal plane polarization pixels on its sensing surface; each focal plane polarization pixel consists of four degrees of polarization. For example, Figure 2 This is a schematic diagram of a focal plane polarized pixel. (Example) Figure 2 As shown, by calling the interface function of the polarization camera, the focal plane polarization pixels of the camera target surface are obtained in a single exposure, that is, the four degrees of polarization distributed under the same pixel.

[0053] S130. Demodulate the outgoing light to obtain the Stokes vector of the outgoing light, and determine the Stokes vector of the incident light based on the Stokes vector of the outgoing light.

[0054] The outgoing light Stokes vector can be understood as the Stokes vector corresponding to the outgoing light, and the incident light Stokes vector can be understood as the Stokes vector corresponding to the incident light. The Stokes vector is a vector used to describe the intensity and linear polarization information of light, and can be composed of three real-valued parameters, typically expressed as... .

[0055] For example, the outgoing light Stokes vector S corresponding to the outgoing light acquired from three exposures. out It can be represented as:

[0056] ;

[0057] in, Let represent the i-th real parameter in the Stokes vector of the outgoing light acquired during the m-th exposure, where i = 1, 2, 3; m = 1, 2, 3. i = 1, 2, 3 represent the three degrees of polarization of the focusing plane polarizer in the spatial domain.

[0058] The representation of the outgoing light Stokes vector is basically consistent with that of the incoming light Stokes vector S corresponding to the outgoing light acquired in three exposures. in It can be represented as:

[0059] ;

[0060] Let represent the i-th real parameter in the incident Stokes vector of the incident light corresponding to the outgoing light acquired during the m-th exposure, where i = 1, 2, 3; m = 1, 2, 3. i = 1, 2, 3 represent the three degrees of polarization of the focusing plane polarizer in the spatial domain.

[0061] Optionally, determining the incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states, and demodulating the outgoing light to obtain the outgoing light Stokes vector, includes: determining the incident light Stokes vector based on the polarization state components of the incident light with multiple wavelengths and multiple polarization states in the spatial domain; and determining the outgoing light Stokes vector based on the polarization pixels of the focal plane.

[0062] Specifically, the incident light Stokes vector is determined based on the polarization state components corresponding to the polarization modulation of the incident light in the spatial domain by the modulation module. The outgoing light Stokes vector is determined based on the focal plane polarization pixels of the camera target surface acquired in each preset exposure.

[0063] S140. Input the incident light Stokes vector and the outgoing light Stokes vector into the optical transmission model to solve for the Müller matrix of the target under test, and decompose the Müller parameter image set from the Müller matrix of the target under test.

[0064] The optical transmission model can be considered a mathematical model describing the optical transmission process. It defines how incident light emitted from the light source interacts with the material and geometry of the object under test, ultimately generating outgoing light from the object's surface or medium towards the sensor. The input parameter of the optical transmission model is the incident light Stokes vector, and the output parameter is the outgoing light Stokes vector. The known parameters are the calibration Müller matrices of the imaging module and the modulation module, while the unknown parameter is the Müller matrix of the target object. The Müller matrix is ​​a real number matrix, and its core function is to completely characterize the ability of an optical element or medium to transform the polarization state of light.

[0065] For example, the optical transmission model can be represented as:

[0066] ;

[0067] in, The Stokes vector of the emitted light. Let denoted as the Stokes vector of the incident light, and MM as the Müller matrix of the transmission medium during the transmission of the incident light to the outgoing light.

[0068] In this disclosure, the transmission medium involved in the transmission process from incident light to outgoing light includes a modulation module, a target under test, and an imaging module. Therefore, the Müller matrix of the transmission medium can be composed of the Müller matrices corresponding to the modulation module, the target under test, and the imaging module, respectively. Thus, the optical transmission model can be expressed as:

[0069] ;

[0070] Among them, MM i For the calibration Müller matrix of the imaging module, MM j The calibration Müller matrix for the modulation module can be calculated and calibrated by detecting standard samples (air, standard optical elements), etc. MM s Let Müller's matrix be the target to be tested.

[0071] Specifically, the incident light Stokes vector and the outgoing light Stokes vector are substituted into the optical transmission model to solve the optical transmission model and obtain the Müller matrix of the target under test; the Müller matrix of the target under test is decomposed and transformed to obtain the Müller parameter image set of the Müller matrix; the Müller parameter image set represents the feature dataset corresponding to each measurement point of the target under test;

[0072] For example, after determining the incident light Stokes vector and the emitted light Stokes vector in S130, the incident light Stokes vector and the emitted light Stokes vector are substituted into the constructed optical transmission model. This is because the input parameters of the optical transmission model... and output parameters Given, and given, the calibration Müller matrix MM of the imaging module. i The calibration Müller matrix MM of the modulation module j Therefore, the unique unknown parameter, namely the Müller matrix MM of the target object, can be obtained by solving. s .

[0073] Because the Müller matrix stores a large amount of physical information in high-dimensional data form, it can be decomposed and transformed into a product or sum of "fundamental" matrices representing different independent physical processes. From these fundamental matrices, parameter images with clear physical meaning can be extracted; these parameters are called Müller parameter images. These Müller parameter images are saved and displayed in grayscale image format, which allows for the analysis and visualization of key data in polarization imaging.

[0074] For example, methods for decomposing and transforming the Müller matrix may include polarization decomposition, differential decomposition, Claude decomposition, and Müller matrix transformation. Müller parameter images may include reflectance maps, dichroism correlation parameter maps (amplitude and direction, etc.), delay correlation parameter maps, and depolarization correlation parameter maps (such as depolarization coefficients and depolarization entropy).

[0075] The Müller matrix obtained by imaging the incident light at each wavelength corresponds to N Müller parameter images. In this embodiment, since the incident light is modulated into a multi-wavelength, multi-polarization state by polarization spectrum, K×N Müller parameter images can be obtained. These K×N Müller parameter images construct a K×N feature dataset at one measurement point of the target. Compared to existing vision systems that acquire one Müller parameter image at a time, i.e., constructing one feature dataset at one measurement point of the target, this significantly expands the image dataset. Furthermore, by encoding and decoding the spectral and polarization information of the incident and emitted light, the system expands from the traditional single light intensity dimension to three information dimensions: light intensity, spectrum, and polarization. This reduces the impact of ambient light on the robot's vision capabilities, enhances the detection capability of small targets, and thus improves the quality of visual imaging.

[0076] The technical solution of this invention involves obtaining incident light with multiple wavelengths and multiple polarization states by performing polarization spectrum modulation on the incident light through a modulation module; the polarization spectrum modulation includes time-division spectral modulation and spatial polarization modulation; the incident light is irradiated onto the surface of the target under test, and the outgoing light with multiple wavelengths and multiple polarization states reflected from the surface of the target under test is collected by an imaging module; the incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the outgoing light Stokes vector is obtained by demodulating the outgoing light; the incident light Stokes vector and the outgoing light Stokes vector are input into the optical transmission model to solve the Müller matrix of the target under test, and the Müller parameter image set is decomposed from the Müller matrix of the target under test. By employing time-division spectral modulation and spatial polarization modulation of the incident light, the number of images acquired in a single camera acquisition action is increased. By decomposing the Müller matrix of the target under test, the acquired image information is expanded and enhanced, thus expanding the information dimension of visual imaging. It extends from the traditional single light intensity dimension to three information dimensions: light intensity, spectrum, and polarization. This can reduce the impact of ambient light on the robot's visual capabilities, enhance the detection capability of small targets, and improve the quality of visual imaging.

[0077] In an optional embodiment, after decomposing the Müller parameter image set from the Müller matrix of the target to be tested, the method further includes: inputting the Müller parameter image set into a target detection model to obtain the feature confidence of each measurement point of the target to be tested; accumulating the feature confidence of the same label for the Müller parameter image set corresponding to each measurement point to obtain the cumulative feature confidence; if the cumulative confidence is greater than the label threshold, labeling the target to be tested and saving the feature data of the target to be tested.

[0078] The object detection model can be understood as a fully trained model for detecting objects. Cumulative feature confidence can be understood as the sum of the feature confidence scores for Müller parameter images with the same label.

[0079] Specifically, the obtained Müller parameter image set is used as the feature dataset and input into the target detection model. The target detection model performs target segmentation and feature recognition on the Müller parameter image set to obtain the feature confidence of each measurement point of the target. For each measurement point, the feature confidence of the Müller parameter images with the same label in the K×N Müller parameter image set is accumulated to obtain the cumulative feature confidence. A manually set label threshold is used as a discriminator to judge the recognition results. The label threshold can be set by empirical values ​​in practical applications. If the cumulative confidence is greater than the label threshold, the target features can be marked in the visualization interface, and key information such as feature type, location, area, and confidence is saved. If the cumulative confidence is less than the label threshold, the next point's Müller parameter image set is judged until all points on the target are judged.

[0080] This disclosure includes a visual interface in the terminal for remote, real-time viewing and control of the inspection robot's vision capabilities. The visual interface can present high-latitude data images, define target features, and label key target information. It can also provide interactive interfaces, such as controlling the rotating motor-driven optical components, setting basic parameters like the polarization camera's sensitivity and exposure time, setting tag threshold parameters, and saving and retrieving key target information.

[0081] The algorithm employs a combined physical system and data-driven approach. It performs target segmentation and recognition on the image set obtained from solving the Müller parameters, while simultaneously setting a preset threshold for the cumulative confidence of the same label. This improves accuracy and provides a function to save key information, facilitating the tracking and processing of detection information.

[0082] Example 2

[0083] Figure 3 This is a schematic diagram of the structure of a robot vision system provided in Embodiment 2 of the present invention. Figure 3 As shown, the robot vision system 200 includes: a modulation module 210, a polarization-independent beam splitter 220, an imaging module 230, and a host computer 240; wherein:

[0084] The modulation module 210 is used to receive the control of the host computer, convert the incident light into incident light with multiple wavelengths and multiple polarization states through a multi-wavelength filter and a focal plane polarizer, and irradiate the surface of the target under test with the incident light.

[0085] The polarization-independent beam splitter 220 is used to separate the multi-wavelength, multi-polarization outgoing light reflected from the surface of the target under test from the main optical path to the imaging module.

[0086] The imaging module 230 is used to collect the outgoing light with multiple wavelengths and multiple polarization states reflected from the surface of the target under test;

[0087] The host computer 240 is used to determine the incident light Stokes vector corresponding to the incident light in the multi-wavelength and multi-polarization state, and to demodulate the outgoing light to obtain the outgoing light Stokes vector; the incident light Stokes vector and the outgoing light Stokes vector are input into the optical transmission model to solve the Müller matrix of the target under test, and the Müller parameter image set is decomposed from the Müller matrix of the target under test.

[0088] This invention provides a robot vision system, comprising: a modulation module, a polarization-independent beam splitter, an imaging module, and a host computer; the modulation module receives control from the host computer, converts incident light into multi-wavelength, multi-polarization incident light through a multi-wavelength filter and a focal plane polarizer, and illuminates the surface of a target under test with the incident light; the polarization-independent beam splitter separates the multi-wavelength, multi-polarization outgoing light reflected from the surface of the target under test from the main optical path to the imaging module; the imaging module collects the multi-wavelength, multi-polarization outgoing light reflected from the surface of the target under test; the host computer determines the incident light Stokes vector corresponding to the multi-wavelength, multi-polarization incident light, and demodulates the outgoing light to obtain the outgoing light Stokes vector; the incident light Stokes vector and the outgoing light Stokes vector are input into an optical transmission model to solve for the Müller matrix of the target under test, and a Müller parameter image set is decomposed from the Müller matrix of the target under test. By employing time-division spectral modulation and spatial polarization modulation of the incident light, the number of images acquired in a single camera acquisition action is increased. By decomposing the Müller matrix of the target under test, the acquired image information is expanded and enhanced, thus expanding the information dimension of visual imaging. It extends from the traditional single light intensity dimension to three information dimensions: light intensity, spectrum, and polarization. This can reduce the impact of ambient light on the robot's visual capabilities, enhance the detection capability of small targets, and improve the quality of visual imaging.

[0089] Optional, Figure 4 This is a schematic diagram of another robot vision system provided in Embodiment 2 of the present invention. Figure 4 As shown, the modulation module 210 includes a multi-wavelength filter 211 and a focal plane polarizer 212;

[0090] The multi-wavelength filter is used to perform time-division spectral modulation on the incident light to obtain multi-wavelength incident light;

[0091] A split-focus plane polarizer is used to perform spatial polarization modulation on incident light of each wavelength separately, so as to obtain incident light with multiple wavelengths and multiple polarization states.

[0092] Optionally, the polarization-independent beam splitter is used to separate the outgoing light reflected from the surface of the target under test from the main optical path to the imaging module.

[0093] Optionally, the imaging module 230 includes a lens group 231 and a polarization camera 232;

[0094] The lens group is used to focus outgoing light with multiple wavelengths and multiple polarization states;

[0095] The polarization camera is used to acquire the focal plane polarization pixels of the emitted light under a preset exposure, and the focal plane polarization pixels are composed of four degrees of polarization.

[0096] Optionally, the host computer 240 includes:

[0097] An incident light Stokes vector determination module is used to determine the incident light Stokes vector based on the polarization state components of the multi-wavelength, multi-polarization incident light in the spatial domain.

[0098] The outgoing light Stokes vector determination module is used to determine the outgoing light Stokes vector based on the focal plane polarization pixel.

[0099] Optionally, the host computer 240 further includes:

[0100] The Müller matrix determination module is used to substitute the incident light Stokes vector and the outgoing light Stokes vector into the optical transmission model, and solve the optical transmission model to obtain the Müller matrix of the target to be measured.

[0101] The Müller parameter image set determination unit is used to decompose and transform the Müller matrix of the target to be measured to obtain the Müller parameter image set of the Müller matrix; the Müller parameter image set represents the feature dataset corresponding to each measurement point of the target to be measured;

[0102] The input parameters of the optical transmission model are the incident light Stokes vector and the output parameters are the outgoing light Stokes vector. The known parameters are the calibration Müller matrix of the imaging module and the calibration Müller matrix of the modulation module, and the unknown parameter is the Müller matrix of the target to be measured.

[0103] Optionally, the host computer 240 controls the rotary motor 250 and drives the filter wheel to switch between multiple wavelength filters in a time-division manner.

[0104] Optionally, the white light source is integrated into a parallel beam through a lens group and an aperture, and the parallel beam is used as the incident light.

[0105] The robot vision system provided in this embodiment of the invention can execute the visual imaging method of the robot provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0106] Example 3

[0107] Figure 5A schematic diagram of a robot 10, which can be used to implement embodiments of the present invention, is shown. The term "robot" is intended to represent various forms of programmable machines capable of performing a range of complex tasks through programming and automatic control, such as inspection robots, industrial robots, and service robots. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0108] like Figure 5 As shown, robot 10 includes at least one processor 11 and a memory, such as read-only memory (ROM) 12, random access memory (RAM) 13, etc., communicatively connected to at least one processor 11. The memory stores computer programs executable by at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer program stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of robot 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. Input / output (I / O) interface 15 is also connected to bus 14.

[0109] Multiple components in robot 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows robot 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0110] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as visual imaging methods for robots.

[0111] In some embodiments, the robot's visual imaging method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the robot 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the robot's visual imaging method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the robot's visual imaging method by any other suitable means (e.g., by means of firmware).

[0112] Figure 6 This is a structural diagram of the robot's vision module. (Example) Figure 6 As shown, the robot's vision module integrates a vision imaging system 200, which includes a modulation module 210 and an imaging module 230. The output of the modulation module 210 emits incident light that illuminates the surface of the target and reflects it to the inlet of the imaging module 230.

[0113] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] In some embodiments, the robot's visual imaging method can be implemented as a computer program, which is implicitly included in a computer program product. When executed by a processor, the computer program implements the robot's visual imaging method of the present invention. The computer program product can be understood as a software product that primarily implements its solution through a computer program. The computer program used to implement the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer program causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a remote machine as a standalone software package, or entirely on a remote machine or server.

[0115] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0116] To provide interaction with a user, the systems and techniques described herein can be implemented on a robot having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the robot. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0117] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0118] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0119] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0120] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method of visual imaging of a robot, characterized by, The application is applied to a robot, comprising: The incident light is modulated by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectrum modulation includes time spectrum modulation and spatial domain polarization modulation; The incident light is irradiated to the surface of the target to be measured, and the surface reflection of the target to be measured is collected by an imaging module; The incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the exit light Stokes vector is calculated by demodulation; The incident light Stokes vector and the exit light Stokes vector are input into an optical transfer model to solve the Mueller matrix of the target to be measured, and the Mueller parameter image set is decomposed from the Mueller matrix of the target to be measured.

2. The method of claim 1, wherein, The incident light is modulated by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectrum modulation includes time spectrum modulation and spatial domain polarization modulation; The incident light is irradiated to the surface of the target to be measured, and the surface reflection of the target to be measured is collected by an imaging module; The incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the exit light Stokes vector is calculated by demodulation; The incident light Stokes vector and the exit light Stokes vector are input into an optical transfer model to solve the Mueller matrix of the target to be measured, and the Mueller parameter image set is decomposed from the Mueller matrix of the target to be measured.

3. The method of claim 1, wherein, The incident light is modulated by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectrum modulation includes time spectrum modulation and spatial domain polarization modulation; The incident light is irradiated to the surface of the target to be measured, and the surface reflection of the target to be measured is collected by an imaging module; The incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the exit light Stokes vector is calculated by demodulation; The incident light Stokes vector and the exit light Stokes vector are input into an optical transfer model to solve the Mueller matrix of the target to be measured, and the Mueller parameter image set is decomposed from the Mueller matrix of the target to be measured.

4. The method of claim 3, wherein, The incident light is modulated by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectrum modulation includes time spectrum modulation and spatial domain polarization modulation; The incident light is irradiated to the surface of the target to be measured, and the surface reflection of the target to be measured is collected by an imaging module; The incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the exit light Stokes vector is calculated by demodulation; 5. The method of claim 1, wherein, The incident light Stokes vector and the exit light Stokes vector are input into an optical transfer model to solve the Mueller matrix of the target to be measured, and the Mueller parameter image set is decomposed from the Mueller matrix of the target to be measured. The incident light is modulated by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectrum modulation includes time spectrum modulation and spatial domain polarization modulation; The incident light is irradiated to the surface of the target to be measured, and the surface reflection of the target to be measured is collected by an imaging module; The incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the exit light Stokes vector is calculated by demodulation; 6. The method of claim 1, wherein, The incident light Stokes vector and the exit light Stokes vector are input into an optical transfer model to solve the Mueller matrix of the target to be measured, and the Mueller parameter image set is decomposed from the Mueller matrix of the target to be measured. The incident light is modulated by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectrum modulation includes time spectrum modulation and spatial domain polarization modulation; The incident light is irradiated to the surface of the target to be measured, and the surface reflection of the target to be measured is collected by an imaging module; The incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the exit light Stokes vector is calculated by demodulation; The incident light Stokes vector and the exit light Stokes vector are input into an optical transfer model to solve the Mueller matrix of the target to be measured, and the Mueller parameter image set is decomposed from the Mueller matrix of the target to be measured. The incident light is modulated by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectrum modulation includes time spectrum modulation and spatial domain polarization modulation; The incident light is irradiated to the surface of the target to be measured, and the surface reflection of the target to be measured is collected by an imaging module; The incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the exit light Stokes vector is calculated by demodulation; The incident light Stokes vector and the exit light Stokes vector are input into an optical transfer model to solve the Mueller matrix of the target to be measured, and the Mueller parameter image set is decomposed from the Mueller matrix of the target to be measured. The incident light is modulated by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectrum modulation includes time spectrum modulation and spatial domain polarization modulation; The incident light is irradiated to the surface of the target to be measured, and the surface reflection of the target to be measured is collected by an imaging module; The incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the exit light Stokes vector is calculated by demodulation; The incident light Stokes vector and the exit light Stokes vector are input into an optical transfer model to solve the Mueller matrix of the target to be measured, and the Mueller parameter image set is decomposed from the Mueller matrix of the target to be measured. The incident light is modulated by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectrum modulation includes time spectrum modulation and spatial domain polarization modulation; The incident light is irradiated to the surface of the target to be measured, and the surface reflection of the target to be measured is collected by an imaging module; The incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the exit light Stokes vector is calculated by demodulation; The incident light Stokes vector and the exit light Stokes vector are input into an optical transfer model to solve the Mueller matrix of the target to be measured, and the Mueller parameter image set is decomposed from the Mueller matrix of the target to be measured. The incident light is modulated by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectrum modulation includes time spectrum modulation and spatial domain polarization modulation; The incident light is irradiated to the surface of the target to be measured, and the surface reflection of the target to be measured is collected by an imaging module; The incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the exit light Stokes vector is calculated by demodulation; The incident light Stokes vector and the exit light Stokes vector are input into an optical transfer model to solve the Mueller matrix of the target to be measured, and the Mueller parameter image set is decomposed from the Mueller matrix of the target to be measured. The incident light is modulated by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectrum modulation includes time spectrum modulation and spatial domain polarization modulation; The incident light is irradiated to the surface of the target to be measured, and the surface reflection of the target to be measured is collected by an imaging module; The incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the exit light Stokes vector is calculated by demodulation; The incident light Stokes vector and the exit light Stokes vector are input into an optical transfer model to solve the Mueller matrix of the target to be measured, and the Mueller parameter image set is decomposed from the Mueller matrix of the target to be measured. The incident light is modulated by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectrum modulation includes time spectrum modulation and spatial domain polarization modulation; The incident light is irradiated to the surface of the target to be measured, and the surface reflection of the target to be measured is collected by an imaging module; The incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the exit light Stokes vector is calculated by demodulation; The incident light Stokes vector and the exit light Stokes vector are input into an optical transfer model to solve the Mueller matrix of the target to be measured, and the Mueller parameter image set is decomposed from the Mueller matrix of the target to be measured. The incident light is modulated by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectrum modulation includes time spectrum modulation and spatial domain polarization modulation; The incident light is irradiated to the surface of the target to be measured, and the surface reflection of the target to be measured is collected by an imaging module; The incident light Stokes vector corresponding to the incident light with multiple wavelengths and multiple polarization states is determined, and the exit light Stokes vector is calculated by demodulation; The incident light Stokes vector and the exit light Stokes vector are input into an optical transfer model to solve the Mueller matrix of the target to be measured, and the Mueller parameter image set is decomposed from the Mueller matrix of the target to be measured. The incident light is modulated by a modulation module to obtain incident light with multiple wavelengths and multiple polarization states; the polarization spectrum modulation includes time spectrum modulation and spatial domain polarization modulation; The incident light is irradiated to the surface of the target to be measured, and the surface reflection of the target to be measured Input the Mueller parameter image set into a target detection model to obtain feature confidence of each measurement point of the target to be measured; For each measurement point, the Mueller parameter image set corresponding to the measurement point is subjected to feature confidence accumulation of the same label to obtain cumulative feature confidence; If the cumulative confidence is greater than a label threshold, the target to be measured is marked and feature data of the target to be measured is saved.

7. A robotic vision system characterized by, Comprise: A modulation module, a polarization-independent beam splitter, an imaging module, and a host computer; The modulation module is configured to receive control from the host computer, convert incident light into multi-wavelength and multi-polarization state incident light through a multi-wavelength filter and a focal plane polarizer, and irradiate the incident light onto the surface of a target to be measured; The polarization-independent beam splitter is configured to separate the multi-wavelength and multi-polarization state outgoing light reflected by the surface of the target to be measured from the main light path to the imaging module; The imaging module is configured to collect the multi-wavelength and multi-polarization state outgoing light reflected by the surface of the target to be measured; The host computer is configured to determine the incident light Stokes vector corresponding to the multi-wavelength and multi-polarization state incident light, calculate the outgoing light Stokes vector by demodulating the outgoing light, input the incident light Stokes vector and the outgoing light Stokes vector into an optical transfer model to solve the Mueller matrix of the target to be measured, and decompose the Mueller parameter image set from the Mueller matrix of the target to be measured.

8. A robot, characterized in that The robot comprises: At least one processor; and a memory connected in communication with the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the visual imaging method of the robot according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to execute the visual imaging method of the robot according to any one of claims 1-6 when executed by the processor.

10. A computer program product, characterised in that, The computer program product comprises a computer program which, when executed by the processor, implements the visual imaging method of the robot according to any one of claims 1-6.

Images