Automatic Information Acquisition Method and Device Based on Multi-Mode Sensors

Abstract

The present invention discloses a method and device for automatically acquiring information based on a multi-mode sensor. The multi-mode sensor is provided with an optical flexible film and an infrared light source that emits light in a preset infrared wavelength band to irradiate the optical flexible film, and the infrared light source is in an on state. The method includes: collecting an image corresponding to a target object based on the multi-mode sensor, and segmenting the image corresponding to the target object to obtain a first image of the target object and a first infrared image of the optical flexible film; analyzing the information of the target object according to the first image; and analyzing the information of the contact force exerted by the target object on the optical flexible film according to the first infrared image. It can be seen that the present invention collects the image of the object and the infrared image of the optical flexible film through a vision-tactile multi-mode sensor that is not affected by electromagnetic interference and has high resolution, and analyzes them simultaneously, which can improve the analysis accuracy and efficiency of the visual information and tactile information of the object.

Description

Technical Field

[0001] The present invention relates to the technical field of sensors, and particularly to a method and device for automatically acquiring information based on a multi-mode sensor. Background Art

[0002] During the processes of disordered grasping, sorting, and assembling by a robot that require fine operations, vision and touch are often used in combination. That is, it is necessary to identify and locate the position of an object through vision and determine the magnitude and angle of the force applied to the object through touch to complete corresponding tasks.

[0003] In practical applications, multiple independent sensors are usually installed on the robot, and the data collected by the multiple independent sensors are analyzed to control the robot to complete corresponding tasks. However, it is found in practice that existing sensors, such as tactile sensors, calculate force based on capacitance and resistance changes and are easily affected by electromagnetic interference, so accurate data cannot be accurately collected, and thus precise control of the robot cannot be achieved. Therefore, it is very important to propose a technical solution for accurately collecting relevant data to achieve precise control of the robot. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and device for automatically acquiring information based on a multi-mode sensor. By using a vision-tactile multi-mode sensor that is not affected by electromagnetic interference and has high resolution to collect an image of an object and an infrared image of an optical flexible film, and analyzing them simultaneously, the accuracy and efficiency of analyzing the visual information and tactile information of the object can be improved, which is beneficial to improving the accuracy and reliability of determining information such as the position, size, and contact force between the object and the multi-mode sensor, and further helps to improve the control accuracy and efficiency of the robot.

[0005] To solve the above technical problem, in a first aspect of the present invention, a method for automatically acquiring information based on a multi-mode sensor is disclosed. The multi-mode sensor is provided with an optical flexible film and an infrared light source that emits light in a preset infrared wavelength band to irradiate the optical flexible film, and the infrared light source is in an on state;

[0006] The method includes:

[0007] Collecting an image corresponding to a target object based on the multi-mode sensor, and segmenting the image corresponding to the target object to obtain a first image of the target object and a first infrared image of the optical flexible film. The first infrared image is an image collected when the target object acts on the optical flexible film;

[0008] Analyzing the information of the target object according to the first image;

[0009] Analyze information on the contact force exerted by the target object on the optical flexible film based on the first infrared image;

[0010] Among them, the information on the target object and the information on the contact force are used to control the machine equipment to perform operations matching the information on the target object and the information on the contact force.

[0011] As an alternative implementation manner, in the first aspect of the present invention, before collecting the image corresponding to the target object based on the multi-mode sensor, the method further includes:

[0012] Perform a calibration operation on the multi-mode sensor to obtain a calibration model of the multi-mode sensor; the calibration model includes a visual calibration model and a tactile calibration model;

[0013] In addition, an optical label is provided in the optical flexible film; the analyzing the information on the contact force received by the multi-mode sensor according to the first infrared image includes:

[0014] Convert each pixel position in the first infrared image into a corresponding first reconstructed pixel position in a preset world coordinate system, and calculate the value of the first reconstructed pixel position based on a pre-determined pixel value formula;

[0015] Based on all the first reconstructed pixel positions and the values of all the first reconstructed pixel positions, determine the first reconstructed image of the first infrared image;

[0016] Obtain the target optical label position information of the optical label according to the first reconstructed image, and input the target optical label position information into the tactile calibration model; the target optical label position information at least includes the optical label position change information of the optical label;

[0017] Obtain the target contact force group output by the tactile calibration model, where the target contact force group includes a contact force intensity and a contact force angle;

[0018] Determine the force intensity and force angle included in the target contact force group as the information on the contact force received by the multi-mode sensor;

[0019] In addition, the analyzing the information on the target object according to the first image includes:

[0020] Convert each pixel position in the first image into a corresponding second reconstructed pixel position in a preset world coordinate system, calculate the value of the second reconstructed pixel position based on a pre-determined pixel value formula, and based on all the second reconstructed pixel positions and the values of all the second reconstructed pixel positions, determine the second reconstructed image of the first image;

[0021] Determine the shape and position information of the target object according to the pixel positions of the texture of the target object in the second reconstructed image, where the texture of the target object includes color texture or grayscale texture;

[0022] Wherein, when the number of the second reconstructed images is 1, the distance between the actual position corresponding to each pixel position in all the pixel positions corresponding to the texture of the target object in the second reconstructed image on the target object and the multi-mode sensor is equal;

[0023] When the number of the second reconstructed images is greater than or equal to 2, the second reconstructed image includes a plurality of first sub-reconstructed images. For each of the first sub-reconstructed images in all the first sub-reconstructed images, the distance between the actual position corresponding to each pixel position in all the pixel positions corresponding to the texture of the target object in the first sub-reconstructed image on the target object and the multi-mode sensor is equal, and for each of the first sub-reconstructed images in all the first sub-reconstructed images, the distance between the actual position corresponding to each pixel position in all the pixel positions corresponding to the texture of the target object in the first sub-reconstructed image on the target object and the multi-mode sensor are not equal.

[0024] As an optional implementation manner, in the first aspect of the present invention, the performing a calibration operation on the multi-mode sensor to obtain a calibration model of the multi-mode sensor includes:

[0025] Determine the tactile detection set of the multi-mode sensor, where the tactile detection set includes a force intensity detection set and a force angle detection set. The force intensity detection set includes a plurality of force intensities, and the force angle detection set includes a plurality of force angles;

[0026] Randomly combine all the force intensities in the force intensity detection set with all the force angles in the force angle detection set to obtain a plurality of contact force groups; each contact force group respectively includes a force intensity and a force angle, and there are no two contact force groups with the same force intensity and force angle in all the contact force groups;

[0027] Control the robot to touch the optical flexible film in sequence according to the force intensity and force angle corresponding to each contact force group, and control the multi-mode sensor to collect the second infrared image of the optical flexible film each time the optical flexible film is touched;

[0028] Determine the position information of the optical label when the optical flexible film is subjected to the contact force group according to the second infrared image corresponding to each contact force group, and obtain a plurality of first optical label position information;

[0029] Determine the optical label position change information corresponding to each contact force group according to the second optical label position information and the first optical label position information corresponding to each contact force group; wherein, the second optical label position information is the position information of the optical label when the optical flexible film is in a non-loaded state;

[0030] Establish an association relationship between each contact force group and the optical label position information corresponding to the contact force group, and construct a tactile calibration model of the multi-mode sensor as the calibration model of the multi-mode sensor based on the association relationship between each contact force group and the optical label position information corresponding to the contact force group in all the contact force groups;

[0031] Wherein, when the number of the optical labels is 1, the optical label position information corresponding to the contact force group is the optical label position change information corresponding to the contact force group, or the optical label position information corresponding to the contact force group is the optical label position change information corresponding to the contact force group and the second optical label position information;

[0032] When the number of the optical labels is greater than or equal to 2, the optical label position information corresponding to the contact force group is the optical label position change information of all the optical labels corresponding to the contact force group, or the optical label position information corresponding to the contact force group is the optical label position change information of all the optical labels corresponding to the contact force group and the second optical label position information.

[0033] As an optional implementation manner, in the first aspect of the present invention, the optical label absorbs the light in the preset infrared wavelength band and does not absorb visible light;

[0034] Wherein, the step of determining the position information of the optical label when the optical flexible film is subjected to the contact force group according to the second infrared image corresponding to each contact force group includes:

[0035] Generate a third reconstructed image corresponding to the contact force group according to the second infrared image corresponding to each contact force group; the pixel value of the pixel position where the optical label is located in the third reconstructed image corresponding to each contact force group is smaller than the pixel values of the other pixel positions except the pixel position where the optical label is located in the third reconstructed image;

[0036] Segment the optical label area corresponding to the contact force group from the third reconstructed image according to the pixel value of the pixel position where the optical label is located in the third reconstructed image corresponding to each contact force group in all the contact force groups;

[0037] Determine the position information of the optical label area corresponding to each contact force group in the third reconstructed image corresponding to the contact force group, as the position information of the optical label corresponding to the contact force group when the optical flexible film is subjected to the contact force group;

[0038] Moreover, the determining the position information of the optical label area corresponding to each contact force group in the third reconstructed image corresponding to the contact force group, as the position information of the optical label corresponding to the contact force group when the optical flexible film is subjected to the contact force group, includes:

[0039] Calculate the position coordinates of the centroid of the optical label area corresponding to each contact force group according to the position information of the optical label area corresponding to the contact force group, and determine the position coordinates of the centroid of the optical label area corresponding to the contact force group as the position information of the optical label corresponding to the contact force group when the optical flexible film is subjected to the contact force group; or,

[0040] Calculate the position coordinates of the geometric center of the optical label area corresponding to each contact force group according to the position information of the optical label area corresponding to the contact force group, and determine the position coordinates of the geometric center of the optical label area corresponding to the contact force group as the position information of the optical label corresponding to the contact force group when the optical flexible film is subjected to the contact force group.

[0041] As an optional implementation manner, in the first aspect of the present invention, the converting each pixel position in the first image into a corresponding second reconstructed pixel position in a preset world coordinate system, calculating the value of the second reconstructed pixel position based on a pre-determined pixel value formula, and determining the second reconstructed image of the first image based on all the second reconstructed pixel positions and the values of all the second reconstructed pixel positions includes:

[0042] Obtain the pixel values of all target pixel positions in the first image, and all the target pixel positions are determined based on a pixel coordinate system;

[0043] Obtain the camera model of the multi-mode sensor, determine the conversion relationship between the preset world coordinate system and the pixel coordinate system according to the camera model, and determine the reconstructed pixel position corresponding to each target pixel position according to the conversion relationship;

[0044] Input the reconstructed pixel position corresponding to each target pixel position into the visual calibration model, and obtain the value output by the visual calibration model as the calibration model output value of the visual calibration model at the reconstructed pixel position corresponding to the target pixel position;

[0045] Input the pixel value of each of the target pixel positions and the calibration model output value of the visual calibration model at the corresponding reconstructed pixel position of the target pixel position into a pre-determined pixel value formula, and inversely solve the pixel value formula to obtain the pixel value of the corresponding reconstructed pixel position of the target pixel position;

[0046] Determine the second reconstructed image of the first image according to the corresponding reconstructed pixel position of each of the target pixel positions and the pixel value of the corresponding reconstructed pixel position of each of the target pixel positions.

[0047] As an optional implementation manner, in the first aspect of the present invention, the performing a calibration operation on the multi-mode sensor to obtain a calibration model of the multi-mode sensor includes:

[0048] Determine a set of visual detection positions of the multi-mode sensor; the set of visual detection positions includes a plurality of visual detection positions, and each of the visual detection positions is any position where the multi-mode sensor can detect the object to be detected;

[0049] Determine all target positions where the calibration light source has not appeared from all the visual detection positions in the set of visual detection positions;

[0050] Control the multi-mode sensor to collect a second image of the calibration light source at all the target positions;

[0051] Preprocess all the second images to obtain a visible light visual calibration model;

[0052] Obtain the structural parameters of the optical flexible film, and determine an infrared light visual calibration model according to the visible light visual calibration model and the structural parameters of the optical flexible film;

[0053] And, the preprocessing all the second images to obtain a visible light visual calibration model includes:

[0054] Control the multi-mode sensor to collect a third image without a light source;

[0055] Perform a denoising operation on all the second images according to the third image to obtain a plurality of fourth visible light images, and determine a visible light visual calibration model according to each pixel position and the value of the pixel position in each of the fourth visible light images among all the fourth visible light images.

[0056] As an optional implementation manner, in the first aspect of the present invention, the pixel value formula is:

[0057]

[0058] Among them, i(u, v) is the pixel value at any pixel position (u, v) in the target image, where the target image includes the first image or the first infrared image, C is a determined constant, and n(u, v) is the imaging noise at the pixel position (u, v).

[0059] When the target image is the first infrared image, z1 and z2 are respectively the minimum distance and the maximum distance between the actual positions corresponding to all pixel positions of the texture of the optical flexible film in the first infrared image on the optical flexible film and the multi-mode sensor, w(x, y, z) is the value at the first reconstructed pixel position (x, y, z) corresponding to the pixel position (u, v) of the first infrared image in the first reconstructed image, and f(x, y, z) is the value of the visual calibration model at the first reconstructed pixel position (x, y, z).

[0060] When the target image is the first image, z1 and z2 are respectively the minimum distance and the maximum distance between the actual positions corresponding to all pixel positions of the texture of the target object in the first image on the target object and the multi-mode sensor, w(x, y, z) is the value at the second reconstructed pixel position (x, y, z) corresponding to the pixel position (u, v) of the first image in the second reconstructed image, and f(x, y, z) is the value of the visual calibration model at the second reconstructed pixel position (x, y, z).

[0061] As an optional implementation manner, in the first aspect of the present invention, the multi-mode sensor includes a micro-optical device, and the optical design objective function of the micro-optical device is obtained by adding the optical design objective functions of lights in each wavelength band multiplied by the weights of the lights in that wavelength band, where all the lights in the wavelength bands include one or more of red light wavelength band light, green light wavelength band light, blue light wavelength band light, and infrared light wavelength band light; and the weight corresponding to the red light wavelength band light is greater than the weights corresponding to each wavelength band light among the other wavelength band lights except the red light wavelength band light among all the wavelength band lights; the optical design objective function of each wavelength band light includes one or more optical indexes of the light in that wavelength band, and the optical design objective function of the micro-optical device is used to design the micro-optical device.

[0062] In addition, a double-sided reflective film is provided on the outer layer of the optical flexible film, the double-sided reflective film does not absorb visible light, the infrared light source is arranged in the internal space of the multi-mode sensor, the inner surface of the double-sided reflective film reflects the preset infrared wavelength band light emitted from the infrared light source back into the internal space of the multi-mode sensor, and the outer surface of the double-sided reflective film reflects the infrared light with the same preset infrared wavelength band from the external environment of the multi-mode sensor back into the external environment.

[0063] The second aspect of the present invention discloses an information automatic acquisition device based on a multi-mode sensor. The multi-mode sensor is provided with an optical flexible film and an infrared light source that emits light in a preset infrared wavelength band to irradiate the optical flexible film, and the infrared light source is in an on state; the device includes:

[0064] An acquisition and processing module, configured to acquire an image corresponding to a target object based on the multi-mode sensor, and segment the image corresponding to the target object to obtain a first image of the target object and a first infrared image of the optical flexible film; the first infrared image is an image acquired when the target object acts on the optical flexible film;

[0065] An analysis module, configured to analyze information of the target object according to the first image;

[0066] The analysis module is further configured to analyze information about the contact force of the target object acting on the optical flexible film according to the first infrared image; wherein, the information of the target object and the information of the contact force are used to control a machine device to perform an operation matching the information of the target object and the information of the contact force.

[0067] As an optional implementation manner, in the second aspect of the present invention, the device further includes:

[0068] A calibration module, configured to perform a calibration operation on the multi-mode sensor before the acquisition and processing module acquires an image corresponding to a target object based on the multi-mode sensor, to obtain a calibration model of the multi-mode sensor; the calibration model includes a visual calibration model and a tactile calibration model;

[0069] And, an optical label is provided in the optical flexible film; the specific manner in which the analysis module analyzes the information about the contact force received by the multi-mode sensor according to the first infrared image includes:

[0070] Convert each pixel position in the first infrared image into a corresponding first reconstructed pixel position in a preset world coordinate system, and calculate the value of the first reconstructed pixel position based on a pre-determined pixel value formula;

[0071] Based on all the first reconstructed pixel positions and the values of all the first reconstructed pixel positions, determine a first reconstructed image of the first infrared image;

[0072] Obtain target optical label position information of the optical label according to the first reconstructed image, and input the target optical label position information into the tactile calibration model; the target optical label position information at least includes optical label position change information of the optical label;

[0073] Obtain the target contact force group output by the tactile calibration model, where the target contact force group includes a contact force intensity and a contact force angle;

[0074] Determine the force intensity and force angle included in the target contact force group as the information of the contact force received by the multi-mode sensor;

[0075] Moreover, the specific manner in which the analysis module analyzes the information of the target object based on the first image includes:

[0076] Convert each pixel position in the first image into a corresponding second reconstructed pixel position in a preset world coordinate system, calculate the value of this second reconstructed pixel position based on a pre-determined pixel value formula, and determine the second reconstructed image of the first image based on all the second reconstructed pixel positions and the values of all the second reconstructed pixel positions;

[0077] Determine the shape and position information of the target object according to the pixel positions of the texture of the target object in the second reconstructed image, where the texture of the target object includes a color texture or a grayscale texture;

[0078] Wherein, when the number of the second reconstructed images is 1, the distance between each actual position corresponding to each pixel position in all the pixel positions corresponding to the texture of the target object in the second reconstructed image on the target object and the multi-mode sensor is equal;

[0079] When the number of the second reconstructed images is greater than or equal to 2, the second reconstructed image includes a plurality of first sub-reconstructed images. The distance between each actual position corresponding to each pixel position in all the pixel positions corresponding to the texture of the target object in each first sub-reconstructed image among all the first sub-reconstructed images on the target object and the multi-mode sensor is equal, and the distances between each actual position corresponding to each pixel position in all the pixel positions corresponding to the texture of the target object in each first sub-reconstructed image among all the first sub-reconstructed images on the target object and the multi-mode sensor are not equal.

[0080] As an optional implementation manner, in the second aspect of the present invention, the specific manner in which the calibration module performs a calibration operation on the multi-mode sensor to obtain the calibration model of the multi-mode sensor includes:

[0081] Determine the tactile detection set of the multi-mode sensor, where the tactile detection set includes a force intensity detection set and a force angle detection set. The force intensity detection set includes a plurality of force intensities, and the force angle detection set includes a plurality of force angles;

[0082] Randomly combine all the force intensities in the force intensity detection set with all the force angles in the force intensity detection set to obtain a plurality of contact force groups; each of the contact force groups respectively includes a force intensity and a force angle, and there are no two contact force groups with the same force intensity and force angle in all the contact force groups;

[0083] Control the robot to touch the optical flexible film in sequence according to the force intensity and force angle corresponding to each contact force group, and control the multi-mode sensor to collect the second infrared image of the optical flexible film each time it is touched;

[0084] Determine the position information of the optical label on the optical flexible film when the optical flexible film is subjected to the contact force group according to the second infrared image corresponding to each contact force group, and obtain a plurality of first optical label position information;

[0085] Determine the optical label position change information corresponding to each contact force group according to the second optical label position information and the first optical label position information corresponding to each contact force group; wherein, the second optical label position information is the position information of the optical label when the optical flexible film is in a non-force state;

[0086] Establish an association relationship between each contact force group and the optical label position information corresponding to the contact force group, and construct a tactile calibration model of the multi-mode sensor based on the association relationship between each contact force group and the optical label position information corresponding to the contact force group in all the contact force groups, as the calibration model of the multi-mode sensor;

[0087] Wherein, when the number of the optical labels is 1, the optical label position information corresponding to the contact force group is the optical label position change information corresponding to the contact force group, or the optical label position information corresponding to the contact force group is the optical label position change information corresponding to the contact force group and the second optical label position information;

[0088] When the number of the optical labels is greater than or equal to 2, the optical label position information corresponding to the contact force group is the optical label position change information of all the optical labels corresponding to the contact force group, or the optical label position information corresponding to the contact force group is the optical label position change information of all the optical labels corresponding to the contact force group and the second optical label position information.

[0089] As an optional implementation manner, in the second aspect of the present invention, the optical label absorbs the light in the preset infrared wavelength band and does not absorb visible light;

[0090] Among them, the way for the calibration module to determine the position information of the optical tag when the optical flexible film is subjected to each contact force group according to the second infrared image corresponding to each contact force group specifically includes:

[0091] Generating a third reconstructed image corresponding to each contact force group according to the second infrared image corresponding to each contact force group; the pixel value of the pixel position where the optical tag is located in the third reconstructed image corresponding to each contact force group is less than the pixel values of other pixel positions except the pixel position where the optical tag is located in this third reconstructed image;

[0092] Segmenting the optical tag area corresponding to each contact force group from this third reconstructed image according to the pixel value of the pixel position where the optical tag is located in the third reconstructed image corresponding to each contact force group in all contact force groups;

[0093] Determining the position information of the optical tag area corresponding to each contact force group in the third reconstructed image corresponding to this contact force group as the position information of the optical tag corresponding to this contact force group when the optical flexible film is subjected to this contact force group;

[0094] Moreover, the way for the calibration module to determine the position information of the optical tag area corresponding to each contact force group in the third reconstructed image corresponding to this contact force group as the position information of the optical tag corresponding to this contact force group when the optical flexible film is subjected to this contact force group specifically includes:

[0095] Calculating the position coordinates of the centroid of the optical tag area corresponding to each contact force group according to the position information of the optical tag area corresponding to each contact force group, and determining the position coordinates of the centroid of the optical tag area corresponding to this contact force group as the position information of the optical tag corresponding to this contact force group when the optical flexible film is subjected to this contact force group; or,

[0096] Calculating the position coordinates of the geometric center of the optical tag area corresponding to each contact force group according to the position information of the optical tag area corresponding to each contact force group, and determining the position coordinates of the geometric center of the optical tag area corresponding to this contact force group as the position information of the optical tag corresponding to this contact force group when the optical flexible film is subjected to this contact force group.

[0097] As an optional implementation manner, in the second aspect of the present invention, the way for the analysis module to convert each pixel position in the first image into a corresponding second reconstructed pixel position in a preset world coordinate system, calculate the value of this second reconstructed pixel position based on a pre-determined pixel value formula, and determine the second reconstructed image of the first image based on all the second reconstructed pixel positions and the values of all the second reconstructed pixel positions specifically includes:

[0098] Obtain the pixel values of all target pixel positions in the first image, and all the target pixel positions are determined based on the pixel coordinate system;

[0099] Obtain the camera model of the multi-mode sensor, determine the conversion relationship between the preset world coordinate system and the pixel coordinate system according to the camera model, and determine the reconstructed pixel position corresponding to each target pixel position according to the conversion relationship;

[0100] Input the reconstructed pixel position corresponding to each target pixel position into the visual calibration model, and obtain the value output by the visual calibration model as the calibration model output value of the visual calibration model at the reconstructed pixel position corresponding to the target pixel position;

[0101] Input the pixel value of each target pixel position and the calibration model output value of the visual calibration model at the reconstructed pixel position corresponding to the target pixel position into a pre-determined pixel value formula, and inversely solve the pixel value formula to obtain the pixel value of the reconstructed pixel position corresponding to the target pixel position;

[0102] Determine the second reconstructed image of the first image according to the reconstructed pixel position corresponding to each target pixel position and the pixel value of the reconstructed pixel position corresponding to each target pixel position.

[0103] As an optional implementation manner, in the second aspect of the present invention, the manner in which the calibration module performs a calibration operation on the multi-mode sensor to obtain the calibration model of the multi-mode sensor specifically includes:

[0104] Determine the visual detection position set of the multi-mode sensor; the visual detection position set includes a plurality of visual detection positions, and each visual detection position is any position where the multi-mode sensor can detect the object to be detected;

[0105] Determine all target positions where the calibration light source has not appeared from all the visual detection positions in the visual detection position set;

[0106] Control the multi-mode sensor to collect second images of the calibration light source at all the target positions;

[0107] Preprocess all the second images to obtain a visible light visual calibration model;

[0108] Obtain the structural parameters of the optical flexible film, and determine an infrared light visual calibration model according to the visible light visual calibration model and the structural parameters of the optical flexible film;

[0109] Moreover, the manner in which the calibration module preprocesses all the second images to obtain a visible light visual calibration model specifically includes:

[0110] Control the multi-mode sensor to collect a third image without a light source;

[0111] Perform a denoising operation on all the second images according to the third image to obtain a plurality of fourth visible light images, and determine a visible light vision calibration model according to each pixel position and the value of each pixel position in each of the fourth visible light images.

[0112] As an optional implementation manner, in the second aspect of the present invention, the pixel value formula is:

[0113]

[0114] where i(u, v) is the pixel value at any pixel position (u, v) in the target image, the target image includes the first image or the first infrared image, C is a determined constant, and n(u, v) is the imaging noise at the pixel position (u, v);

[0115] When the target image is the first infrared image, z1 and z2 are respectively the minimum distance and the maximum distance between the actual positions corresponding to all pixel positions of the texture of the optical flexible film in the first infrared image on the optical flexible film and the multi-mode sensor, w(x, y, z) is the value at the first reconstructed pixel position (x, y, z) corresponding to the pixel position (u, v) of the first infrared image in the first reconstructed image, and f(x, y, z) is the value of the vision calibration model at the first reconstructed pixel position (x, y, z);

[0116] When the target image is the first image, z1 and z2 are respectively the minimum distance and the maximum distance between the actual positions corresponding to all pixel positions of the texture of the target object in the first image on the target object and the multi-mode sensor, w(x, y, z) is the value at the second reconstructed pixel position (x, y, z) corresponding to the pixel position (u, v) of the first image in the second reconstructed image, and f(x, y, z) is the value of the vision calibration model at the second reconstructed pixel position (x, y, z).

[0117] As an alternative embodiment, in the second aspect of the present invention, the multimode sensor includes a micro-optical device, and the optical design objective function of the micro-optical device is obtained by adding the optical design objective functions of the light in each wavelength band multiplied by the weight of the light in that wavelength band, where all the light in the wavelength bands includes one or more of red-wavelength light, green-wavelength light, blue-wavelength light, and infrared-wavelength light; and the weight corresponding to the red-wavelength light is greater than the weight corresponding to each wavelength band of the other wavelength bands of light except the red-wavelength light among all the wavelength bands of light; the optical design objective function of each wavelength band of light includes one or more optical indicators of the light in that wavelength band, and the optical design objective function of the micro-optical device is used to design the micro-optical device;

[0118] In addition, a double-sided reflective film is provided on the outer layer of the optical flexible film. The double-sided reflective film does not absorb visible light. The infrared light source is disposed in the internal space of the multimode sensor. The inner surface of the double-sided reflective film reflects the preset infrared-wavelength light emitted from the infrared light source back into the internal space of the multimode sensor, and the outer surface of the double-sided reflective film reflects the infrared light with the same preset infrared-wavelength as that from the external environment of the multimode sensor back into the external environment.

[0119] The third aspect of the present invention discloses another information automatic acquisition device based on a multimode sensor. The device includes:

[0120] A memory storing executable program code;

[0121] A processor coupled to the memory;

[0122] The processor calls the executable program code stored in the memory and executes the information automatic acquisition method based on the multimode sensor disclosed in the first aspect of the present invention.

[0123] The fourth aspect of the present invention discloses a computer-readable storage medium. The computer storage medium stores computer instructions, which are used to execute the information automatic acquisition method based on the multimode sensor disclosed in the first aspect of the present invention when the computer instructions are called.

[0124] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0125] In an embodiment of the present invention, the multi-mode sensor is provided with an optical flexible film and an infrared light source that emits light in a preset infrared wavelength band to irradiate the optical flexible film, and the infrared light source is in an on state; the method includes: collecting an image corresponding to a target object based on the multi-mode sensor, and segmenting the image corresponding to the target object to obtain a first image of the target object and a first infrared image of the optical flexible film, where the first infrared image is an image collected when the target object acts on the optical flexible film; analyzing information of the target object according to the first image; analyzing information of the contact force of the target object acting on the optical flexible film according to the first infrared image; where the information of the target object and the information of the contact force are used to control a machine device to perform an operation matching the information of the target object and the information of the contact force. It can be seen that the present invention collects an image of an object and an infrared image of an optical flexible film through a visual-tactile multi-mode sensor that is not affected by electromagnetic interference and has high resolution, and analyzes them simultaneously, which can improve the analysis accuracy and efficiency of the visual information and tactile information of the object, thereby facilitating improving the determination accuracy and reliability of information such as the position and size of the object and the contact force between the object and the multi-mode sensor, and further contributing to improving the control accuracy and efficiency of the robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0126] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0127] Figure 1 It is a schematic flowchart of a method for automatically obtaining information based on a multi-mode sensor disclosed in an embodiment of the present invention;

[0128] Figure 2 It is a schematic structural diagram of a multi-mode sensor disclosed in an embodiment of the present invention;

[0129] Figure 3 It is a schematic structural diagram of an optical flexible film disclosed in an embodiment of the present invention;

[0130] Figure 4 It is a schematic structural diagram of a micro-optical device disclosed in an embodiment of the present invention;

[0131] Figure 5 It is a schematic path diagram of two groups of optical paths in a multi-mode sensor disclosed in an embodiment of the present invention;

[0132] Figure 6 It is a schematic flowchart of another method for automatically obtaining information based on a multi-mode sensor disclosed in an embodiment of the present invention;

[0133] Figure 7 It is a schematic structural diagram of an information automatic acquisition device based on a multi - mode sensor disclosed in an embodiment of the present invention;

[0134] Figure 8 It is a schematic structural diagram of another information automatic acquisition device based on a multi - mode sensor disclosed in an embodiment of the present invention;

[0135] Figure 9 It is a schematic structural diagram of yet another information automatic acquisition device based on a multi - mode sensor disclosed in an embodiment of the present invention. Specific embodiments

[0136] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present invention.

[0137] The terms "first", "second", etc. in the specification and claims of the present invention and the above - mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non - exclusive inclusion. For example, a process, method, device, product or terminal comprising a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or terminals.

[0138] Referring to "embodiments" herein means that specific features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present invention. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0139] The present invention discloses an information automatic acquisition method and device based on a multi - mode sensor. By collecting the images of an object and the infrared images of an optical flexible film through a vision - tactile multi - mode sensor that is not affected by electromagnetic interference and has high resolution, and analyzing them simultaneously, the accuracy and efficiency of analyzing the visual information and tactile information of the object can be improved. Thus, it is beneficial to improve the accuracy and reliability of determining information such as the position, size of the object, and the contact force between the object and the multi - mode sensor, and further helps to improve the control accuracy and efficiency of the robot. The following will be described in detail respectively.

[0140] Embodiment 1

[0141] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of a method for automatically acquiring information based on a multi-mode sensor disclosed in an embodiment of the present invention. Among them, Figure 1 the described method for automatically acquiring information based on a multi-mode sensor can be applied to a device for automatically acquiring information based on a multi-mode sensor, where the device includes any one of a robot control device, a robot control system, a robot control server, a robot control platform, etc., and the device has a corresponding multi-mode sensor. As Figure 1 shown, the method for automatically acquiring information based on a multi-mode sensor may include the following operations:

[0142] 101. Collect an image corresponding to a target object based on a multi-mode sensor, and segment the image corresponding to the target object to obtain a first image of the target object and a first infrared image of an optical flexible film.

[0143] In an embodiment of the present invention, as Figure 2 shown, the multi-mode sensor includes an optical flexible film (also referred to as an optical flexible film structure), an infrared light source (which can be a NIR light source, i.e., a near-infrared light source), a micro-optical device (also referred to as a flat-type micro-optical device), a photoelectric sensor, a PCB (including a system-on-chip SoC), a support structure 1, a support structure 2, and a visible light supplementary light source (also referred to as a VIS light source). Among them, the PCB (including the system-on-chip SoC) is an operation platform.

[0144] Among them, the infrared light source emits light in a preset infrared wavelength band to irradiate the optical flexible film, and the infrared light source is in an on state when the multi-mode sensor is in a working state; among them, the light in the preset infrared wavelength band can be near-infrared light with a wavelength of 1μm to 0.7μm, or infrared light with other wavelengths, and the wavelength includes 500μm to 1μm, which is not limited in the present invention. Among them, a silicon-based image sensor can be used when the infrared light wavelength is 1μm to 0.7μm; a non-silicon-based image sensor can be used when the infrared light wavelength is higher than 1μm. In an embodiment of the present invention, the first infrared image is an image collected when the target object acts on the optical flexible film, and the first image of the target object is a visible light image of the target object.

[0145] In an embodiment of the present invention, as Figure 3 shown, the optical flexible film includes an optical label (which can be a NIR absorption optical label, i.e., a near-infrared light absorption optical label), a double-sided reflective film (which can be a NIR high-reflection film, i.e., a near-infrared high-reflection film), an optical flexible layer, a NIR AR coating, and a transparent substrate.

[0146] Among them, the double-sided reflective film is transparent to visible light. The double-sided reflective film is disposed on the outer layer of the optical flexible film. The outer surface of the double-sided reflective film reflects the light in a preset infrared wavelength band from outside the sensor back into the environment. At the same time, the inner surface of the double-sided reflective film highly reflects the light in the preset infrared wavelength band from inside the sensor. Further, the light in the preset infrared wavelength band from inside the sensor is generated by an infrared light source, and at this time, the infrared light source is disposed inside the sensor. It can be seen that by reflecting the external infrared light through the double-sided reflective film, the influence of the external light in the preset infrared wavelength band on the sensor can be reduced, and the accuracy of collecting the infrared image of the optical flexible film can be improved, thereby contributing to improving the accuracy of subsequent analysis. At the same time, since the optical flexible film is transparent to visible light, the visible light reflected by the target object and the infrared light reflected by the optical flexible film can simultaneously pass through the optical flexible film and enter the inside of the multi-mode sensor for imaging, which helps the multi-mode sensor to share the imaging device and simplifies the structure of the sensor. Optionally, the double-sided reflective film can be a high-reflection film with a reflection efficiency greater than or equal to a preset reflection efficiency.

[0147] In an embodiment of the present invention, in order to reduce the interference of the infrared light source on visible light imaging, if the infrared light source is a wide-spectrum LED, it is necessary to ensure that its spectrum is cut off outside the sensitive spectral window of the red pixel unit (for example, the spectrum of the infrared light source is cut off above 700 nm). This requirement can be achieved by selecting a narrow-spectrum infrared LED light-emitting chip or adding a single-sided optical filter (long-pass edge optical filter) that blocks infrared light and transmits visible light.

[0148] In an embodiment of the present invention, as Figure 4 shown, the micro-optical device includes a VIS-NIR band-pass optical filter layer, a surface microstructure, and an optical substrate. Among them, the optical design objective function of the micro-optical device is obtained by adding the optical design objective functions of each wavelength band of light multiplied by the weight of that wavelength band of light, and the weight corresponding to the red light wavelength band is greater than the weight corresponding to each wavelength band of light among other wavelength bands of light except the red light wavelength band among all wavelength bands of light. Among them, all wavelength bands of light include one or more of red light wavelength band, green light wavelength band, blue light wavelength band, and infrared light wavelength band. Among them, the optical design objective function of each wavelength band of light includes one or more optical indicators of that wavelength band of light. For example, if the optical indicators used in the optical design objective function are related to imaging, they can be one or more optical indicators defined based on the optical transfer function (point spread function PSF / modulation transfer function MTF) or the quality of the image.

[0149] It should be noted that, in order to improve the optical effect of the optical system, representative wavelengths required are usually selected during the optical design process to establish an optical design objective function. For example, when designing the optical design objective function for an RGB vision sensor in the visible light band, generally a higher weight is given to the wavelength within the green light range (such as 550 nm). When a multi-mode sensor needs to work under visible light and infrared light simultaneously, a higher weight can be given to the wavelength within the red light range (such as 700 nm) when designing the optical design objective function. Among them, if the weight of the light in a specific wavelength band is high, the optical index corresponding to the light in this specific wavelength band is the best among all wavelength bands of light. For example, when the optical index of a micro-optical device is chromatic aberration, if the weight of red light is 1 and that of blue light is 0, then the chromatic aberration of red light in the image obtained through the micro-optical device is less than that of blue light; if the weight of red light is 0.5 and that of blue light is 0.5, then the chromatic aberration of red light and blue light in the image obtained through the micro-optical device will be relatively close. It can be seen that allocating weights for the light in each wavelength band in the optical design objective function according to the actual situation and determining the optical design objective function based on the determined weights helps to design a micro-optical device that matches the actual situation according to the determined optical design objective function.

[0150] In an embodiment of the present invention, as Figure 5 shown, the optoelectronic sensor can be a VIS-NIR optoelectronic sensor. Each basic pixel unit in this sensor includes four optoelectronic conversion sub-pixels, which are respectively responsive to red light, green light, blue light, and near-infrared light. Each optoelectronic conversion sub-pixel converts the light in the wavelength band corresponding to this optoelectronic conversion sub-pixel received into a digital signal. As Figure 5 shown, the first group of optical paths of the multi-mode sensor is that a VIS fill light source (set when fill light is required and can be not set when fill light is not required) emits visible light to irradiate the target object, and the light reflected by the target object passes through the optical flexible film and then passes through the micro-optical device and irradiates on the optoelectronic sensor; the second group of optical paths of the multi-mode sensor is that an NIR light source emits light in a preset infrared wavelength band to irradiate the optical flexible film, and the light in the preset infrared wavelength band reflected by the optical flexible film passes through the micro-optical device and irradiates on the optoelectronic sensor; the optoelectronic sensor converts the light received from the two groups of optical paths into digital signals.

[0151] 102. Analyze the information of the target object according to the first image.

[0152] In an embodiment of the present invention, the information of the target object can be at least one of the shape, size, and position of the target object.

[0153] 103. Analyze the information of the contact force exerted by the target object on the optical flexible film according to the first infrared image.

[0154] In the embodiments of the present invention, the information of the target object and the information of the contact force are used to control the machine device to perform operations matching the information of the target object and the information of the contact force.

[0155] It can be seen that the present invention collects the image of the object and the infrared image of the optical flexible film through a vision-tactile multi-mode sensor that is not affected by electromagnetic interference and has high resolution, and analyzes them simultaneously, which can improve the analysis accuracy and efficiency of the visual information and tactile information of the object. Therefore, it is beneficial to improve the determination accuracy and reliability of information such as the position, size of the object, and the contact force between the object and the multi-mode sensor, and further helps to improve the control accuracy and efficiency of the robot.

[0156] Embodiment 2

[0157] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of another method for automatically obtaining information based on a multi-mode sensor disclosed in the embodiments of the present invention. Among them, Figure 2 the method for automatically obtaining information based on a multi-mode sensor described can be applied to a device for automatically obtaining information based on a multi-mode sensor. Among them, the device includes any one of a robot control device, a robot control system, a robot control server, a robot control platform, etc., and the device has a corresponding multi-mode sensor. As Figure 2 shown, the method for automatically obtaining information based on a multi-mode sensor may include the following operations:

[0158] 201. Perform a calibration operation on the multi-mode sensor to obtain a calibration model of the multi-mode sensor.

[0159] In the embodiments of the present invention, the calibration model of the multi-mode sensor includes a vision calibration model and a tactile calibration model.

[0160] In the embodiments of the present invention, as an optional implementation manner, optical tags are provided in the optical flexible film; wherein, performing a calibration operation on the multi-mode sensor to obtain a calibration model of the multi-mode sensor includes:

[0161] Determine the tactile detection set of the multi-mode sensor. The tactile detection set includes a force intensity detection set and a force angle detection set. The force intensity detection set includes multiple force intensities, and the force angle detection set includes multiple force angles;

[0162] Randomly combine all the force intensities in the force intensity detection set with all the force angles in the force intensity detection set to obtain multiple contact force groups; each contact force group contains a force intensity and a force angle respectively, and there are no two contact force groups with the same force intensity and force angle in all the contact force groups;

[0163] Control the robot to touch the optical flexible film in sequence according to the force intensity and force angle corresponding to each contact force group, and control the multi-mode sensor to collect the second infrared image of the optical flexible film each time the optical flexible film is touched;

[0164] Determine the position information of the optical label on the optical flexible film when the optical flexible film is subjected to the contact force group according to the second infrared image corresponding to each contact force group, and obtain a plurality of first optical label position information;

[0165] Determine the position change information of the optical label corresponding to each contact force group according to the second optical label position information and the first optical label position information corresponding to each contact force group; wherein, the second optical label position information is the position information of the optical label when the optical flexible film is in a non-force state;

[0166] Establish an association relationship between each contact force group and the optical label position information corresponding to the contact force group, and construct a tactile calibration model of the multi-mode sensor based on the association relationship between each contact force group and the optical label position information corresponding to the contact force group in all contact force groups, as the calibration model of the multi-mode sensor.

[0167] In this optional embodiment, all the force intensities in the tactile detection set of the multi-mode sensor are within the force intensity detection range of the multi-mode sensor, and all the force angles in the tactile detection set are within the force angle detection range of the multi-mode sensor; the force intensity detection range and the force angle detection range can be determined according to the parameters designed by the sensor itself. In this optional embodiment, the minimum and maximum values of the number of all contact force groups can be set according to the parameters designed by the sensor itself or the requirements of the application scenario.

[0168] In this optional embodiment, when the number of optical labels is 1, the optical label position information corresponding to the contact force group is the optical label position change information corresponding to the contact force group, or the optical label position information corresponding to the contact force group is the optical label position change information corresponding to the contact force group and the second optical label position information; when the number of optical labels is greater than or equal to 2, the optical label position information corresponding to the contact force group is the optical label position change information of all the optical labels corresponding to the contact force group, or the optical label position information corresponding to the contact force group is the optical label position change information of all the optical labels corresponding to the contact force group and the second optical label position information.

[0169] In this optional embodiment, when the number of optical labels is greater than or equal to 2, the optical label position change information of all the optical labels corresponding to a certain contact force group can be the optical label position change information of all the optical labels, or can be simplified to the optical label position change information of a preset number of adjacent optical labels according to the actual situation.

[0170] In this alternative embodiment, when the number of optical tags is greater than or equal to 2, the arrangement of all optical tags is a dot array, which can be a grid-like distribution or an irregular distribution.

[0171] Optionally, according to the second optical tag position information and the first optical tag position information corresponding to each contact force group, determine the optical tag position change information corresponding to each contact force group, including:

[0172] Subtract the second optical tag position information from the first optical tag position information corresponding to each contact force group to obtain the optical tag position change information corresponding to this contact force group.

[0173] It can be seen that this alternative embodiment provides a method for performing a calibration operation. By applying different forces to the optical flexible film provided with optical tags, images of optical tags corresponding to different forces can be obtained, and the relationship between the force and the optical tag position can also be determined based on these images of optical tags and the force. As a tactile calibration model representing the actual tactile information acquisition situation of the multi-mode sensor, it helps to analyze the collected images according to the tactile calibration model subsequently and improve the accuracy and effectiveness of the subsequent force analysis.

[0174] In the embodiment of the present invention, as another alternative implementation, perform a calibration operation on the multi-mode sensor to obtain a calibration model of the multi-mode sensor, including:

[0175] Determine the set of visual detection positions of the multi-mode sensor; the set of visual detection positions includes multiple visual detection positions, and each visual detection position is any position where the multi-mode sensor can detect the object to be detected;

[0176] Determine all target positions where the calibration light source has not appeared from all the visual detection positions in the set of visual detection positions;

[0177] Control the multi-mode sensor to collect the second images of the calibration light source at all target positions;

[0178] Preprocess all the second images to obtain a visible light vision calibration model;

[0179] Obtain the structural parameters of the optical flexible film, and determine the infrared light vision calibration model according to the visible light vision calibration model and the structural parameters of the optical flexible film.

[0180] In this alternative embodiment, when the z-axis of the preset world coordinate system is the distance direction between the multi-mode sensor and the target object, each visual detection position in the preset world coordinate system can have the same position on the x-axis and y-axis, but different positions on the z-axis. In this alternative embodiment, the vision calibration model is an optical transfer function.

[0181] Among them, according to the visible light vision calibration model and the structural parameters of the optical flexible film, the infrared light vision calibration model can be determined. The optical transfer function that can cover the thickness distance of the optical flexible film on the z-axis can be approximately obtained through the optical transfer function on the surface of the optical flexible film and a simple linear model, and used as the infrared light vision calibration model.

[0182] It can be seen that this optional embodiment provides another method for performing the calibration operation. By collecting images of the light source at different positions and analyzing to obtain the vision calibration model that represents the actual vision information acquisition of the multi-mode sensor, it helps to analyze the collected images according to the vision calibration model later and improve the accuracy and effectiveness of analyzing the images of the target object.

[0183] Optionally, preprocess all the second images to obtain the visible light vision calibration model, including:

[0184] Control the multi-mode sensor to collect the third image without a light source;

[0185] Perform a denoising operation on all the second images according to the third image to obtain multiple fourth visible light images, and determine the visible light vision calibration model according to each pixel position and the value of each pixel position in each of all the fourth visible light images.

[0186] Among them, performing a denoising operation on all the second images according to the third image can be: subtracting the pixel value at each pixel position in the third image from the pixel value at each pixel position in each second image.

[0187] It can be seen that this optional embodiment can perform a denoising operation on the images, which helps to construct a vision calibration model according to the denoised images later, thereby improving the accuracy of the vision calibration model.

[0188] 202. Based on the multi-mode sensor, collect the image corresponding to the target object, and segment the image corresponding to the target object to obtain the first image of the target object and the first infrared image of the optical flexible film.

[0189] 203. Analyze the information of the target object according to the first image.

[0190] 204. Analyze the information of the contact force exerted by the target object on the optical flexible film according to the first infrared image.

[0191] In the embodiments of the present invention, for other descriptions of steps 202-step 204, please refer to the detailed descriptions of steps 101-step 103 in Embodiment 1, and the embodiments of the present invention will not be repeated here.

[0192] It can be seen that the present invention collects the image of an object and the infrared image of an optical flexible film through a vision-tactile multi-mode sensor that is not affected by electromagnetic interference and has high resolution, and analyzes them simultaneously, which can improve the accuracy and efficiency of the analysis of the visual information and tactile information of the object, thereby facilitating the improvement of the accuracy and reliability of determining information such as the position, size of the object, and the contact force between the object and the multi-mode sensor, and further contributing to the improvement of the control accuracy and efficiency of the robot; the present invention performs a calibration operation before collecting the image to obtain a calibration model that represents the actual use of the multi-mode sensor, which helps to analyze the collected image according to the actual model obtained by calibration, can improve the accuracy of the analysis, and thus helps to improve the control accuracy of the robot based on the collected data.

[0193] In an alternative embodiment, analyzing the information of the contact force received by the multi-mode sensor according to the first infrared image includes:

[0194] Converting each pixel position in the first infrared image into a corresponding first reconstructed pixel position in a preset world coordinate system, and calculating the value of the first reconstructed pixel position based on a pre-determined pixel value formula;

[0195] Based on all the first reconstructed pixel positions and the values of all the first reconstructed pixel positions, determining a first reconstructed image of the first infrared image;

[0196] Obtaining target optical label position information of the optical label according to the first reconstructed image, and inputting the target optical label position information into a tactile calibration model; the target optical label position information includes at least the optical label position change information of the optical label;

[0197] Obtaining a target contact force group output by the tactile calibration model, where the target contact force group includes a contact force intensity and a contact force angle;

[0198] Determining the force intensity and force angle included in the target contact force group as the information of the contact force received by the multi-mode sensor.

[0199] In this alternative embodiment, the preset world coordinate system is established in the use environment of the multi-mode sensor to describe the positions of the multi-mode sensor and the target object.

[0200] It can be seen that this alternative embodiment determines the reconstructed image based on the pixel value formula, obtains the position information of the optical label through the reconstructed image, and determines the information of the contact force received by the sensor through the position information of the optical label and the tactile calibration model, and can obtain the information of the force based on image analysis. Compared with the sensor that analyzes the force based on capacitance and resistance, the resolution is further improved, and it is not affected by electromagnetic interference, so that the accuracy of the data collected by the sensor can be further improved.

[0201] In another alternative embodiment, analyzing the information of the target object according to the first image includes:

[0202] Convert each pixel position in the first image into a corresponding second reconstructed pixel position in a preset world coordinate system, calculate the value of the second reconstructed pixel position based on a pre-determined pixel value formula, and determine the second reconstructed image of the first image based on all the second reconstructed pixel positions and the values of all the second reconstructed pixel positions;

[0203] Determine the shape and position information of the target object according to the pixel positions of the texture of the target object in the second reconstructed image, where the texture of the target object includes a color texture or a grayscale texture.

[0204] In this optional embodiment, when the number of second reconstructed images is 1, the distance between each actual position corresponding to each pixel position on the target object and the multi-modal sensor among all the pixel positions corresponding to the texture of the target object in the second reconstructed image is equal; when the number of second reconstructed images is greater than or equal to 2, the second reconstructed image includes multiple first sub-reconstructed images, and the distance between each actual position corresponding to each pixel position on the target object and the multi-modal sensor among all the pixel positions corresponding to the texture of the target object in each first sub-reconstructed image among all the first sub-reconstructed images is equal, and the distances between each actual position corresponding to each pixel position on the target object and the multi-modal sensor among all the pixel positions corresponding to the texture of the target object in each first sub-reconstructed image among all the first sub-reconstructed images are not equal.

[0205] Among them, determining the position information of the target object can be based on the pixel positions of the texture of the target object in the second reconstructed image. For example, when the pixel position of a certain texture of the target object is (x0, y0, z0), this means that the actual coordinates of the corresponding part of the target object of this texture in the preset world coordinate system are also (x0, y0, z0). When the representative coordinates of the multi-modal sensor in the preset world coordinate system are (x1, y1, z1), the distance S can be calculated through the formula Calculate the distance S.

[0206] It can be seen that this optional embodiment determines the reconstructed image based on the pixel value formula and determines the shape and position information of the target object according to the reconstructed image, which helps to control the robot to perform operations according to the shape and position information of the target object subsequently.

[0207] In another optional embodiment, the optical tag absorbs light in a preset infrared wavelength band and does not absorb visible light; among them, determining the position information of the optical tag when the optical flexible film is subjected to the contact force group according to the second infrared image corresponding to each contact force group includes:

[0208] Generate a third reconstructed image corresponding to each contact force group based on the second infrared image corresponding to each contact force group; the pixel value at the pixel position of the optical label in the third reconstructed image corresponding to each contact force group is less than the pixel values at other pixel positions in the third reconstructed image except for the pixel position of the optical label.

[0209] Segment the optical label region corresponding to each contact force group from the third reconstructed image according to the pixel value at the pixel position of the optical label in the third reconstructed image corresponding to each contact force group among all contact force groups.

[0210] Determine the position information of the optical label region corresponding to each contact force group in the third reconstructed image corresponding to this contact force group as the position information of the optical label corresponding to this contact force group when the optical flexible film is subjected to this contact force group.

[0211] In this optional embodiment, the local adaptive threshold (Adaptive Threshold) segmentation method can be used to segment the optical label region from the image. Among them, at least one optical label is included in each optical label region.

[0212] It can be seen that after generating the reconstructed image corresponding to the contact force group in this optional embodiment, by determining the position information of the optical label region corresponding to the contact force group in the reconstructed image as the position information of the corresponding optical label when the optical flexible film is subjected to the contact force group, the determination accuracy and reliability of this position information can be improved, which in turn helps to establish the association relationship between the contact force and the position information of the optical label by combining the contact force information subsequently.

[0213] In another optional embodiment, determining the position information of the optical label region corresponding to each contact force group in the third reconstructed image corresponding to this contact force group as the position information of the optical label corresponding to this contact force group when the optical flexible film is subjected to this contact force group includes:

[0214] Calculate the position coordinates of the centroid of the optical label region corresponding to each contact force group according to the position information of the optical label region corresponding to each contact force group, and determine the position coordinates of the centroid of the optical label region corresponding to this contact force group as the position information of the optical label corresponding to this contact force group when the optical flexible film is subjected to this contact force group; or,

[0215] Calculate the position coordinates of the geometric center of the optical label region corresponding to each contact force group according to the position information of the optical label region corresponding to each contact force group, and determine the position coordinates of the geometric center of the optical label region corresponding to this contact force group as the position information of the optical label corresponding to this contact force group when the optical flexible film is subjected to this contact force group.

[0216] It can be seen that this alternative embodiment determines the position information of the optical tag based on the centroid or geometric center of the optical tag area, which can improve the accuracy and efficiency of determining the position of the optical tag, thereby contributing to the accuracy and efficiency of subsequently determining the contact force based on the position of the optical tag.

[0217] In yet another alternative embodiment, converting each pixel position in the first image into a corresponding reconstructed position in a preset world coordinate system, calculating the value of the reconstructed position based on a pre-determined pixel value formula, and determining a second reconstructed image of the first image based on all the reconstructed positions and the values of all the reconstructed positions, includes:

[0218] Obtaining the pixel values of all target pixel positions in the first image, and all the target pixel positions are determined based on the pixel coordinate system;

[0219] Obtaining the camera model of the multi-mode sensor, determining the conversion relationship between the preset world coordinate system and the pixel coordinate system according to the camera model, and determining the corresponding reconstructed pixel position of each target pixel position according to the conversion relationship;

[0220] Inputting the reconstructed pixel position corresponding to each target pixel position into the visual calibration model, and obtaining the value output by the visual calibration model as the calibration model output value of the visual calibration model at the reconstructed pixel position corresponding to the target pixel position;

[0221] Inputting the pixel value of each target pixel position and the calibration model output value of the visual calibration model at the reconstructed pixel position corresponding to the target pixel position into the pre-determined pixel value formula, and inversely solving the pixel value formula to obtain the pixel value of the reconstructed pixel position corresponding to the target pixel position;

[0222] Determining the second reconstructed image of the first image according to the reconstructed pixel position corresponding to each target pixel position and the pixel value of the reconstructed pixel position corresponding to each target pixel position.

[0223] It can be seen that this alternative embodiment determines the corresponding relationship between the pixel positions of the image and the reconstructed image based on the model conversion relationship, and obtains the pixel value of the pixel position of the reconstructed image through the inverse solution of the pixel value formula, which can improve the efficiency and accuracy of obtaining the reconstructed image based on the image, and contribute to subsequently analyzing the information of the target object according to the reconstructed image.

[0224] In yet another alternative embodiment, the pixel value formula can be:

[0225]

[0226] where i(u, v) is the pixel value at any pixel position (u, v) in the target image, the target image includes the first image or the first infrared image, C is the determined constant, and n(u, v) is the imaging noise at the pixel position (u, v);

[0227] When the target image is the first infrared image, z1 and z2 are respectively the minimum distance and the maximum distance between the actual positions corresponding to all pixel positions of the texture of the optical flexible film in the first infrared image on the optical flexible film and the multi-mode sensor, w(x, y, z) is the value at the first reconstructed pixel position (x, y, z) corresponding to the pixel position (u, v) of the first infrared image in the first reconstructed image, and f(x, y, z) is the value of the visual calibration model at the first reconstructed pixel position (x, y, z);

[0228] When the target image is the first image, z1 and z2 are respectively the minimum distance and the maximum distance between the actual positions corresponding to all pixel positions of the texture of the target object in the first image on the target object and the multi-mode sensor, w(x, y, z) is the value at the second reconstructed pixel position (x, y, z) corresponding to the pixel position (u, v) of the first image in the second reconstructed image, and f(x, y, z) is the value of the visual calibration model at the second reconstructed pixel position (x, y, z).

[0229] It can be seen that this optional embodiment provides a pixel value formula, which helps to determine the pixel value of the pixel position of the reconstructed image according to the pixel value formula subsequently, and improves the efficiency and accuracy of determining the pixel value of the pixel position of the reconstructed image.

[0230] Embodiment III

[0231] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of an information automatic acquisition device based on a multi-mode sensor disclosed in an embodiment of the present invention. Among them, Figure 7 The described information automatic acquisition device based on a multi-mode sensor may include any one of a robot control device, a robot control system, a robot control server, a robot control platform, etc., and the device has a corresponding multi-mode sensor. As Figure 7 shown, the multi-mode sensor is provided with an optical flexible film and an infrared light source that emits light in a preset infrared wavelength band to irradiate the optical flexible film, and the infrared light source is in an on state; the information automatic acquisition device based on a multi-mode sensor may include:

[0232] An acquisition and processing module 301, configured to acquire an image corresponding to a target object based on the multi-mode sensor and segment the image corresponding to the target object to obtain a first image of the target object and a first infrared image of the optical flexible film; the first infrared image is an image acquired when the target object acts on the optical flexible film;

[0233] An analysis module 302, configured to analyze the information of the target object according to the first image;

[0234] The analysis module 302 is further configured to analyze information about the contact force exerted by the target object on the optical flexible film based on the first infrared image; wherein, the information about the target object and the contact force is used to control the machine device to perform operations matching the information about the target object and the contact force.

[0235] It can be seen that in the embodiments of the present invention, the visual-tactile multi-mode sensor that is not affected by electromagnetic interference and has high resolution is used to collect the image of the object and the infrared image of the optical flexible film, and analyze them simultaneously, which can improve the accuracy and efficiency of the analysis of the visual information and tactile information of the object, thereby facilitating the improvement of the accuracy and reliability of the determination of information such as the position and size of the object and the contact force between the object and the multi-mode sensor, and further contributing to the improvement of the control accuracy and efficiency of the robot.

[0236] In an optional embodiment, as Figure 8 shown, the device further includes:

[0237] A calibration module 303, configured to perform a calibration operation on the multi-mode sensor before the acquisition and processing module 301 acquires the image corresponding to the target object based on the multi-mode sensor, so as to obtain a calibration model of the multi-mode sensor; the calibration model includes a visual calibration model and a tactile calibration model;

[0238] In addition, an optical label is provided in the optical flexible film; the specific manner in which the analysis module 302 analyzes the information about the contact force received by the multi-mode sensor based on the first infrared image includes:

[0239] Convert each pixel position in the first infrared image into a corresponding first reconstructed pixel position in a preset world coordinate system, and calculate the value of the first reconstructed pixel position based on a pre-determined pixel value formula;

[0240] Based on all the first reconstructed pixel positions and the values of all the first reconstructed pixel positions, determine the first reconstructed image of the first infrared image;

[0241] Obtain the target optical label position information of the optical label according to the first reconstructed image, and input the target optical label position information into the tactile calibration model; the target optical label position information includes at least the optical label position change information of the optical label;

[0242] Obtain the target contact force group output by the tactile calibration model, and the target contact force group includes the contact force intensity and the contact force angle;

[0243] Determine the force intensity and force angle included in the target contact force group as the information about the contact force received by the multi-mode sensor;

[0244] In addition, the specific manner in which the analysis module 302 analyzes the information about the target object based on the first image includes:

[0245] Convert each pixel position in the first image to a corresponding second reconstructed pixel position in a preset world coordinate system, calculate the value of the second reconstructed pixel position based on a pre-determined pixel value formula, and determine the second reconstructed image of the first image based on all the second reconstructed pixel positions and the values of all the second reconstructed pixel positions;

[0246] Determine the shape and position information of the target object according to the pixel positions of the texture of the target object in the second reconstructed image, where the texture of the target object includes a color texture or a grayscale texture;

[0247] Wherein, when the number of the second reconstructed images is 1, the distance between each actual position corresponding to each pixel position on the target object among all the pixel positions corresponding to the texture of the target object in the second reconstructed image and the multi-mode sensor is equal;

[0248] When the number of the second reconstructed images is greater than or equal to 2, the second reconstructed image includes a plurality of first sub-reconstructed images. Among all the first sub-reconstructed images, the distance between each actual position corresponding to each pixel position on the target object among all the pixel positions corresponding to the texture of the target object in each first sub-reconstructed image and the multi-mode sensor is equal, and the distances between each actual position corresponding to each pixel position on the target object among all the pixel positions corresponding to the texture of the target object in each first sub-reconstructed image and the multi-mode sensor in all the first sub-reconstructed images are not equal.

[0249] It can be seen that in this optional embodiment, the reconstructed image is determined based on the pixel value formula, the position information of the optical tag is obtained through the reconstructed image, and the information of the contact force received by the sensor is determined through the position information of the optical tag and the tactile calibration model. The information of the force can be obtained based on image analysis, which further improves the resolution compared with the sensor that analyzes the force based on capacitance and resistance and is not affected by electromagnetic interference, thereby further improving the accuracy of the data collected by the sensor; the reconstructed image is determined based on the pixel value formula, and the shape and position information of the target object are determined according to the reconstructed image, which helps to control the robot to execute operations according to the shape and position information of the target object subsequently.

[0250] In another optional embodiment, as Figure 8 shown, the specific manner in which the calibration module 303 performs a calibration operation on the multi-mode sensor to obtain the calibration model of the multi-mode sensor includes:

[0251] Determine the tactile detection set of the multi-mode sensor. The tactile detection set includes a force intensity detection set and a force angle detection set. The force intensity detection set includes a plurality of force intensities, and the force angle detection set includes a plurality of force angles;

[0252] Randomly combine all the force intensities in the force intensity detection set with all the force angles in the force intensity detection set to obtain multiple contact force groups; each contact force group contains a force intensity and a force angle respectively, and there are no two contact force groups with the same force intensity and force angle in all the contact force groups;

[0253] Control the robot to touch the optical flexible film in sequence according to the force intensity and force angle corresponding to each contact force group, and control the multi-mode sensor to collect the second infrared image of the optical flexible film every time it is touched;

[0254] Determine the position information of the optical label on the optical flexible film when the optical flexible film is subjected to the contact force group according to the second infrared image corresponding to each contact force group, and obtain multiple first optical label position information;

[0255] According to the second optical label position information and the first optical label position information corresponding to each contact force group, determine the optical label position change information corresponding to each contact force group; wherein, the second optical label position information is the position information of the optical label when the optical flexible film is in a non-force state;

[0256] Establish the association relationship between each contact force group and the optical label position information corresponding to the contact force group, and construct the tactile calibration model of the multi-mode sensor based on the association relationship between each contact force group and the optical label position information corresponding to the contact force group in all the contact force groups, as the calibration model of the multi-mode sensor;

[0257] Wherein, when the number of optical labels is 1, the optical label position information corresponding to the contact force group is the optical label position change information corresponding to the contact force group, or the optical label position information corresponding to the contact force group is the optical label position change information corresponding to the contact force group and the second optical label position information;

[0258] When the number of optical labels is greater than or equal to 2, the optical label position information corresponding to the contact force group is the optical label position change information of all the optical labels corresponding to the contact force group, or the optical label position information corresponding to the contact force group is the optical label position change information of all the optical labels corresponding to the contact force group and the second optical label position information.

[0259] It can be seen that this optional embodiment provides a method for performing a calibration operation. By applying different forces to the optical flexible film provided with optical labels, images of optical labels corresponding to different forces can be obtained, and the relationship between the force and the optical label position can also be determined according to these images of optical labels and the force, as the tactile calibration model representing the actual tactile information acquisition situation of the multi-mode sensor, which helps to analyze the collected images according to the tactile calibration model subsequently and improve the accuracy and effectiveness of subsequent force analysis.

[0260] In yet another alternative embodiment, as Figure 8 shown, the optical tag absorbs light in a preset infrared wavelength band and does not absorb visible light;

[0261] Among them, the manner in which the calibration module 303 determines the position information of the optical tag on the optical flexible film when the optical flexible film is subjected to the contact force group according to the second infrared image corresponding to each contact force group specifically includes:

[0262] Generating a third reconstructed image corresponding to the contact force group according to the second infrared image corresponding to each contact force group; the pixel value of the pixel position where the optical tag is located in the third reconstructed image corresponding to each contact force group is less than the pixel values of the other pixel positions in the third reconstructed image except for the pixel position where the optical tag is located;

[0263] Segmenting the optical tag region corresponding to the contact force group from the third reconstructed image according to the pixel value of the pixel position where the optical tag is located in the third reconstructed image corresponding to each contact force group in all contact force groups;

[0264] Determining the position information of the optical tag region corresponding to each contact force group in the third reconstructed image corresponding to the contact force group as the position information of the optical tag corresponding to the contact force group when the optical flexible film is subjected to the contact force group;

[0265] And, the manner in which the calibration module 303 determines the position information of the optical tag region corresponding to each contact force group in the third reconstructed image corresponding to the contact force group as the position information of the optical tag corresponding to the contact force group when the optical flexible film is subjected to the contact force group specifically includes:

[0266] Calculating the position coordinates of the centroid of the optical tag region corresponding to the contact force group according to the position information of the optical tag region corresponding to each contact force group, and determining the position coordinates of the centroid of the optical tag region corresponding to the contact force group as the position information of the optical tag corresponding to the contact force group when the optical flexible film is subjected to the contact force group; or,

[0267] Calculating the position coordinates of the geometric center of the optical tag region corresponding to the contact force group according to the position information of the optical tag region corresponding to each contact force group, and determining the position coordinates of the geometric center of the optical tag region corresponding to the contact force group as the position information of the optical tag corresponding to the contact force group when the optical flexible film is subjected to the contact force group.

[0268] It can be seen that after generating the reconstructed image corresponding to the contact force group in this optional embodiment, by determining the position information of the optical label area corresponding to the contact force group in the reconstructed image as the position information of the corresponding optical label when the optical flexible film is subjected to the contact force group, the accuracy and reliability of the determination of this position information can be improved, which in turn helps to establish the correlation between the contact force and the position information of the optical label by combining the contact force information subsequently; determining the position information of the optical label according to the centroid or geometric center of the optical label area can improve the accuracy and efficiency of determining the position of the optical label, thereby helping to improve the accuracy and efficiency of determining the contact force according to the position of the optical label subsequently.

[0269] In another optional embodiment, as Figure 8 shown, the analysis module 302 converts the position of each pixel in the first image into the corresponding second reconstructed pixel position in the preset world coordinate system, and calculates the value of the second reconstructed pixel position based on the pre-determined pixel value formula. Based on all the second reconstructed pixel positions and the values of all the second reconstructed pixel positions, the method for determining the second reconstructed image of the first image specifically includes:

[0270] Obtain the pixel values of all target pixel positions in the first image, and all the target pixel positions are determined based on the pixel coordinate system;

[0271] Obtain the camera model of the multi-mode sensor, determine the conversion relationship between the preset world coordinate system and the pixel coordinate system according to the camera model, and determine the reconstructed pixel position corresponding to each target pixel position according to the conversion relationship;

[0272] Input the reconstructed pixel position corresponding to each target pixel position into the visual calibration model, and obtain the value output by the visual calibration model as the calibration model output value of the visual calibration model at the reconstructed pixel position corresponding to the target pixel position;

[0273] Input the pixel value of each target pixel position and the calibration model output value of the visual calibration model at the reconstructed pixel position corresponding to the target pixel position into the pre-determined pixel value formula, and inversely solve the pixel value formula to obtain the pixel value of the reconstructed pixel position corresponding to the target pixel position;

[0274] Determine the second reconstructed image of the first image according to the reconstructed pixel position corresponding to each target pixel position and the pixel value of the reconstructed pixel position corresponding to each target pixel position.

[0275] It can be seen that this optional embodiment determines the corresponding relationship between the pixel positions of the image and the reconstructed image based on the model conversion relationship, and obtains the pixel value of the pixel position of the reconstructed image by inversely solving the pixel value formula, which can improve the efficiency and accuracy of obtaining the reconstructed image based on the image, and helps to analyze the information of the target object according to the reconstructed image subsequently.

[0276] In yet another alternative embodiment, as Figure 8 shown, the way for the calibration module 303 to perform a calibration operation on the multi-mode sensor and obtain the calibration model of the multi-mode sensor specifically includes:

[0277] Determine the set of visual detection positions of the multi-mode sensor; the set of visual detection positions includes multiple visual detection positions, and each visual detection position is any position where the multi-mode sensor can detect the object to be detected;

[0278] Determine all target positions where the calibration light source has not appeared from all the visual detection positions in the set of visual detection positions;

[0279] Control the multi-mode sensor to collect second images of the calibration light source at all target positions;

[0280] Preprocess all the second images to obtain a visible light vision calibration model;

[0281] Obtain the structural parameters of the optical flexible film, and determine the infrared light vision calibration model according to the visible light vision calibration model and the structural parameters of the optical flexible film;

[0282] Moreover, the way for the calibration module 303 to preprocess all the second images to obtain a visible light vision calibration model specifically includes:

[0283] Control the multi-mode sensor to collect third images without a light source;

[0284] Perform a denoising operation on all the second images according to the third images to obtain multiple fourth visible light images, and determine the visible light vision calibration model according to each pixel position and the value of the pixel position in each of the fourth visible light images among all the fourth visible light images.

[0285] It can be seen that this alternative embodiment provides another method for performing a calibration operation. By collecting images of the light source at different positions and analyzing to obtain a vision calibration model that represents the actual vision information acquisition situation of the multi-mode sensor, it helps to analyze the collected images according to the vision calibration model subsequently, improving the accuracy and effectiveness of analyzing the images of the target object; it can perform a denoising operation on the images, which helps to construct a vision calibration model according to the denoised images subsequently, thereby improving the accuracy of the vision calibration model.

[0286] In yet another alternative embodiment, as Figure 8 shown, the pixel value formula is:

[0287]

[0288] Wherein, i(u, v) is the pixel value at any pixel position (u, v) in the target image, the target image includes the first image or the first infrared image, C is the determined constant, and n(u, v) is the imaging noise at the pixel position (u, v).

[0289] When the target image is the first infrared image, z1 and z2 are respectively the minimum distance and the maximum distance between the actual positions corresponding to all pixel positions of the texture of the optical flexible film in the first infrared image on the optical flexible film and the multi-mode sensor, w(x, y, z) is the value at the first reconstructed pixel position (x, y, z) corresponding to the pixel position (u, v) of the first infrared image in the first reconstructed image, and f(x, y, z) is the value of the visual calibration model at the first reconstructed pixel position (x, y, z).

[0290] When the target image is the first image, z1 and z2 are respectively the minimum distance and the maximum distance between the actual positions corresponding to all pixel positions of the texture of the target object in the first image on the target object and the multi-mode sensor, w(x, y, z) is the value at the second reconstructed pixel position (x, y, z) corresponding to the pixel position (u, v) of the first image in the second reconstructed image, and f(x, y, z) is the value of the visual calibration model at the second reconstructed pixel position (x, y, z).

[0291] It can be seen that this optional embodiment provides a pixel value formula, which helps to determine the pixel value of the pixel position of the reconstructed image according to the pixel value formula subsequently, and improves the efficiency and accuracy of determining the pixel value of the pixel position of the reconstructed image.

[0292] Embodiment Four

[0293] Please refer to Figure 9 , Figure 9 which is a schematic structural diagram of another information automatic acquisition device based on a multi-mode sensor disclosed in the embodiments of the present invention. As Figure 9 shown, the information automatic acquisition device based on a multi-mode sensor may include:

[0294] A memory 401 storing executable program code;

[0295] A processor 402 coupled to the memory 401;

[0296] The processor 402 calls the executable program code stored in the memory 401 and executes the steps in the information automatic acquisition method based on a multi-mode sensor described in Embodiment One or Embodiment Two of the present invention.

[0297] Embodiment Five

[0298] An embodiment of the present invention discloses a computer-storable medium. The computer storage medium stores computer instructions, which, when called, are used to execute the steps in the information automatic acquisition method based on a multi-mode sensor described in Embodiment 1 or Embodiment 2 of the present invention.

[0299] Embodiment 6

[0300] An embodiment of the present invention discloses a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute the steps in the information automatic acquisition method based on a multi-mode sensor described in Embodiment 1 or Embodiment 2.

[0301] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0302] Through the specific descriptions of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, and the storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disk memories, tape memories, or any other computer-readable medium capable of carrying or storing data.

[0303] Finally, it should be noted that: The information automatic acquisition method and device based on a multi-mode sensor disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, rather than limiting them; Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An information automatic acquisition method based on a multi-mode sensor, characterized in that The multi-mode sensor is provided with an optical flexible film and an infrared light source that emits light in a preset infrared wavelength band to irradiate the optical flexible film, and the infrared light source is in an on state; An optical tag is provided in the optical flexible film; the method includes: Performing a calibration operation on the multi-mode sensor to obtain a calibration model of the multi-mode sensor; the calibration model includes a visual calibration model and a tactile calibration model; Collecting an image corresponding to a target object based on the multi-mode sensor, and segmenting the image corresponding to the target object to obtain a first image of the target object and a first infrared image of the optical flexible film, where the first infrared image is an image collected when the target object acts on the optical flexible film; the first image is a visible light image of the target object; Analyzing information of the target object according to the first image; Converting each pixel position in the first infrared image into a corresponding first reconstructed pixel position in a preset world coordinate system, and calculating the value of the first reconstructed pixel position based on a preset pixel value formula; Determining a first reconstructed image of the first infrared image based on all the first reconstructed pixel positions and the values of all the first reconstructed pixel positions; Obtaining target optical tag position information of the optical tag according to the first reconstructed image, and inputting the target optical tag position information into the tactile calibration model; the target optical tag position information at least includes optical tag position change information of the optical tag; Obtaining a target contact force group output by the tactile calibration model, where the target contact force group includes a contact force intensity and a contact force angle; Determining the force intensity and force angle included in the target contact force group as information on the contact force received by the multi-mode sensor; Wherein, the information of the target object and the information of the contact force are used to control a machine device to perform an operation matching the information of the target object and the information of the contact force; And, the analyzing the information of the target object according to the first image includes: Converting each pixel position in the first image into a corresponding second reconstructed pixel position in a preset world coordinate system, and calculating the value of the second reconstructed pixel position based on a preset pixel value formula, and determining a second reconstructed image of the first image based on all the second reconstructed pixel positions and the values of all the second reconstructed pixel positions; Determining the shape and position information of the target object according to the pixel positions of the texture of the target object in the second reconstructed image, where the texture of the target object includes a color texture or a gray-scale texture; Wherein, when the number of the second reconstructed images is 1, the distance between each actual position corresponding to each pixel position of the texture of the target object in the second reconstructed image on the target object and the multi-mode sensor is equal; When the number of the second reconstructed images is greater than or equal to 2, the second reconstructed images include a plurality of first sub-reconstructed images. For each of the first sub-reconstructed images among all the first sub-reconstructed images, the distances between the actual positions corresponding to each pixel position in the texture of the target object on the target object and the multi-modal sensor are equal. For each of the first sub-reconstructed images among all the first sub-reconstructed images, the distances between the actual positions corresponding to each pixel position in the texture of the target object on the target object and the multi-modal sensor are not equal.

2. The information automatic acquisition method based on a multi-mode sensor according to claim 1, wherein Performing a calibration operation on the multi-modal sensor to obtain a calibration model of the multi-modal sensor includes: Determining a tactile detection set of the multi-modal sensor, where the tactile detection set includes a force intensity detection set and a force angle detection set. The force intensity detection set includes a plurality of force intensities, and the force angle detection set includes a plurality of force angles; Randomly combining all the force intensities in the force intensity detection set with all the force angles in the force angle detection set to obtain a plurality of contact force groups; each contact force group respectively includes a force intensity and a force angle, and there are no two contact force groups with the same force intensity and force angle among all the contact force groups; Controlling the robot to sequentially touch the optical flexible film according to the force intensity and force angle corresponding to each contact force group, and controlling the multi-modal sensor to collect a second infrared image of the optical flexible film each time the optical flexible film is touched; Determining the position information of the optical tag when the optical flexible film is subjected to the contact force group according to the second infrared image corresponding to each contact force group, to obtain a plurality of first optical tag position information; Determining the optical tag position change information corresponding to each contact force group according to the second optical tag position information and the first optical tag position information corresponding to each contact force group; where the second optical tag position information is the position information of the optical tag when the optical flexible film is in a non-force state; Establishing an association relationship between each contact force group and the optical tag position information corresponding to the contact force group, and constructing a tactile calibration model of the multi-modal sensor based on the association relationship between each contact force group and the optical tag position information corresponding to the contact force group among all the contact force groups, as the calibration model of the multi-modal sensor; Where, when the number of the optical tags is 1, the optical tag position information corresponding to the contact force group is the optical tag position change information corresponding to the contact force group, or the optical tag position information corresponding to the contact force group is the optical tag position change information corresponding to the contact force group and the second optical tag position information; When the number of the optical tags is greater than or equal to 2, the optical tag position information corresponding to the contact force group is the optical tag position change information of all the optical tags corresponding to the contact force group, or the optical tag position information corresponding to the contact force group is the optical tag position change information of all the optical tags corresponding to the contact force group and the second optical tag position information.

3. The information automatic acquisition method based on a multi-mode sensor according to claim 2, wherein The optical tag absorbs light in the preset infrared wavelength band and does not absorb visible light; Wherein, determining the position information of the optical tag when the optical flexible film is subjected to the contact force group according to the second infrared image corresponding to each contact force group includes: Generating a third reconstructed image corresponding to the contact force group according to the second infrared image corresponding to each contact force group; the pixel value of the pixel position where the optical tag is located in the third reconstructed image corresponding to each contact force group is less than the pixel values of the other pixel positions except the pixel position where the optical tag is located in the third reconstructed image; Segmenting the optical tag region corresponding to the contact force group from the third reconstructed image according to the pixel value of the pixel position where the optical tag is located in the third reconstructed image corresponding to each contact force group among all the contact force groups; Determining the position information of the optical tag region corresponding to each contact force group in the third reconstructed image corresponding to the contact force group as the position information of the optical tag corresponding to the contact force group when the optical flexible film is subjected to the contact force group; And, determining the position information of the optical tag region corresponding to each contact force group in the third reconstructed image corresponding to the contact force group as the position information of the optical tag corresponding to the contact force group when the optical flexible film is subjected to the contact force group includes: Calculating the position coordinates of the centroid of the optical tag region corresponding to the contact force group according to the position information of the optical tag region corresponding to each contact force group, and determining the position coordinates of the centroid of the optical tag region corresponding to the contact force group as the position information of the optical tag corresponding to the contact force group when the optical flexible film is subjected to the contact force group; or, Calculating the position coordinates of the geometric center of the optical tag region corresponding to the contact force group according to the position information of the optical tag region corresponding to each contact force group, and determining the position coordinates of the geometric center of the optical tag region corresponding to the contact force group as the position information of the optical tag corresponding to the contact force group when the optical flexible film is subjected to the contact force group.

4. The information automatic acquisition method based on a multi-mode sensor according to claim 1, wherein Converting each pixel position in the first image into a corresponding second reconstructed pixel position in a preset world coordinate system, calculating the value of the second reconstructed pixel position based on a pre-determined pixel value formula, and determining the second reconstructed image of the first image based on all the second reconstructed pixel positions and the values of all the second reconstructed pixel positions includes: Obtaining the pixel values of all target pixel positions in the first image, and all the target pixel positions are determined based on a pixel coordinate system; Obtain the camera model of the multi-mode sensor, determine the conversion relationship between the preset world coordinate system and the pixel coordinate system according to the camera model, and determine the reconstructed pixel position corresponding to each target pixel position according to the conversion relationship; Input the reconstructed pixel position corresponding to each target pixel position into the visual calibration model, and obtain the value output by the visual calibration model as the calibration model output value of the visual calibration model at the reconstructed pixel position corresponding to the target pixel position; Input the pixel value of each target pixel position and the calibration model output value of the visual calibration model at the reconstructed pixel position corresponding to the target pixel position into the pre-determined pixel value formula, and inversely solve the pixel value formula to obtain the pixel value at the reconstructed pixel position corresponding to the target pixel position; Determine the second reconstructed image of the first image according to the reconstructed pixel position corresponding to each target pixel position and the pixel value at the reconstructed pixel position corresponding to each target pixel position.

5. The method for automatically obtaining information based on a multi-mode sensor according to claim 1, characterized in that, The performing a calibration operation on the multi-mode sensor to obtain a calibration model of the multi-mode sensor includes: Determine the visual detection position set of the multi-mode sensor; the visual detection position set includes a plurality of visual detection positions, and each visual detection position is any position where the multi-mode sensor can detect the object to be detected; Determine all target positions where the calibration light source has not appeared from all the visual detection positions in the visual detection position set; Control the multi-mode sensor to collect second images of the calibration light source at all the target positions; Preprocess all the second images to obtain a visible light visual calibration model; Obtain the structural parameters of the optical flexible film, and determine an infrared light visual calibration model according to the visible light visual calibration model and the structural parameters of the optical flexible film; And, the preprocessing all the second images to obtain a visible light visual calibration model includes: Control the multi-mode sensor to collect a third image without a light source; Perform a denoising operation on all the second images according to the third image to obtain a plurality of fourth visible light images, and determine the visible light visual calibration model according to each pixel position and the value at the pixel position of each fourth visible light image in all the fourth visible light images.

6. The information automatic acquisition method based on a multi-mode sensor according to claim 1, characterized in that The pixel value formula is: where i(u, v) is the pixel value at any pixel position (u, v) in the target image, the target image includes the first image or the first infrared image, C is a determined constant, and n(u, v) is the imaging noise at the pixel position (u, v); When the target image is the first infrared image, z1 and z2 are respectively the minimum distance and the maximum distance between the actual positions corresponding to all pixel positions of the texture of the optical flexible film in the first infrared image on the optical flexible film and the multi-mode sensor, w(x, y, z) is the value at the first reconstructed pixel position (x, y, z) corresponding to the pixel position (u, v) of the first infrared image in the first reconstructed image, and f(x, y, z) is the value of the visual calibration model at the first reconstructed pixel position (x, y, z); When the target image is the first image, z1 and z2 are respectively the minimum distance and the maximum distance between the actual positions corresponding to all pixel positions of the texture of the target object in the first image on the target object and the multi-mode sensor, w(x, y, z) is the value at the second reconstructed pixel position (x, y, z) corresponding to the pixel position (u, v) of the first image in the second reconstructed image, and f(x, y, z) is the value of the visual calibration model at the second reconstructed pixel position (x, y, z).

7. The method for automatically obtaining information based on a multi-mode sensor according to claim 1, characterized in that, The multi-mode sensor includes a micro-optical device, and the optical design objective function of the micro-optical device is obtained by adding the optical design objective functions of light in each wavelength band multiplied by the weights of the light in that wavelength band. Among them, all the wavelength bands of light include one or more of red-wavelength light, green-wavelength light, blue-wavelength light, and infrared-wavelength light; and the weight corresponding to the red-wavelength light is greater than the weights corresponding to each wavelength band of the other wavelength bands of light except the red-wavelength light among all the wavelength bands of light; the optical design objective function of each wavelength band of light includes one or more optical indicators of that wavelength band of light, and the optical design objective function of the micro-optical device is used to design the micro-optical device; Moreover, a double-sided reflective film is provided on the outer layer of the optical flexible film. The double-sided reflective film does not absorb visible light. The infrared light source is disposed in the internal space of the multi-mode sensor. The inner surface of the double-sided reflective film reflects the preset infrared wavelength band light emitted from the infrared light source back into the internal space of the multi-mode sensor, and the outer surface of the double-sided reflective film reflects the infrared light with the same preset infrared wavelength band from the external environment of the multi-mode sensor back into the external environment.

8. An information automatic acquisition device based on a multi-mode sensor, characterized in that The multi-mode sensor is provided with an optical flexible film and an infrared light source that emits preset infrared wavelength band light to irradiate the optical flexible film, and the infrared light source is in an on state; An optical label is provided in the optical flexible film; The device includes: A calibration module, configured to perform a calibration operation on the multi-mode sensor to obtain a calibration model of the multi-mode sensor; the calibration model includes a visual calibration model and a tactile calibration model; The acquisition and processing module is used to acquire an image corresponding to a target object based on the multi-mode sensor, and segment the image corresponding to the target object to obtain a first image of the target object and a first infrared image of the optical flexible film; the first infrared image is an image acquired when the target object acts on the optical flexible film; the first image is a visible light image of the target object; The analysis module is used to analyze the information of the target object according to the first image; The analysis module is further configured to convert each pixel position in the first infrared image into a corresponding first reconstructed pixel position in a preset world coordinate system, and calculate the value of the first reconstructed pixel position based on a preset pixel value formula; based on all the first reconstructed pixel positions and the values of all the first reconstructed pixel positions, determine a first reconstructed image of the first infrared image; obtain target optical label position information of the optical label according to the first reconstructed image, and input the target optical label position information into the tactile calibration model; the target optical label position information at least includes optical label position change information of the optical label; obtain a target contact force group output by the tactile calibration model, where the target contact force group includes a contact force intensity and a contact force angle; determine the force intensity and force angle included in the target contact force group as information about the contact force received by the multi-mode sensor; Wherein, the information of the target object and the information of the contact force are used to control the machine device to perform an operation matching the information of the target object and the information of the contact force; Moreover, the specific manner in which the analysis module analyzes the information of the target object according to the first image includes: Convert each pixel position in the first image into a corresponding second reconstructed pixel position in a preset world coordinate system, and calculate the value of the second reconstructed pixel position based on a preset pixel value formula. Based on all the second reconstructed pixel positions and the values of all the second reconstructed pixel positions, determine a second reconstructed image of the first image; Determine the shape and position information of the target object according to the pixel positions of the texture of the target object in the second reconstructed image, where the texture of the target object includes a color texture or a gray-scale texture; Wherein, when the number of the second reconstructed images is 1, the distance between each actual position corresponding to each pixel position of the texture of the target object in the second reconstructed image on the target object and the multi-mode sensor is equal; When the number of the second reconstructed images is greater than or equal to 2, the second reconstructed images include a plurality of first sub-reconstructed images. For each of the first sub-reconstructed images among all the first sub-reconstructed images, the distances between the actual positions corresponding to each pixel position among all the pixel positions corresponding to the texture of the target object on the target object and the multi-mode sensor are equal, and the distances between the actual positions corresponding to each pixel position among all the pixel positions corresponding to the texture of the target object on the target object and the multi-mode sensor in each of the first sub-reconstructed images among all the first sub-reconstructed images are not equal.

9. An information automatic acquisition device based on a multi-mode sensor, characterized in that, The device includes: a memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory and executes the method for automatically obtaining information based on a multi-mode sensor according to any one of claims 1-7.

Images