A material identification method and system

Abstract

The present invention provides a material recognition method and system, belonging to the field of optical information processing technology. The method includes: obtaining depth and active infrared reflection intensity data of an object to be identified and extracting the object's posture information, the obtained data including a depth map, an intensity map, and a point cloud map; performing contour extraction and pixel depth and intensity value extraction on the depth map and the intensity map respectively; adding the contour extraction, pixel depth and intensity value extraction to the point cloud map to calculate the horizontal and vertical tilt angles associated with each pixel of the object to be identified; inputting the horizontal and vertical tilt angles, along with the extracted spatial position and depth value of the pixel point, as input data values ​​of a neural network into a pre-trained optimized neural network model, fitting a fitted intensity value for each pixel point, and outputting the fitting result for identifying the material of the object to be identified. This method improves the accuracy of object recognition in different spatial positions and different surface geometries.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical information processing, and in particular relates to a material recognition method and system. Background Art

[0002] With the current development of fields such as artificial intelligence and optical information processing, the application of machine vision to target detection, object recognition and three-dimensional scene reconstruction has become a hot research field. This technology has been widely used in scenarios such as transportation, military, agriculture and industrial production. In this field, one of the more important tasks is to judge the material of the target object.

[0003] Image data fusion is an effective method for improving judgment accuracy. This technique combines images acquired by different sensor types to generate robust and information-rich images, which is more conducive to subsequent image processing. Among the different data fusion combinations, thermal infrared and visible light (TIR-V) fusion technology has many advantages. Their signals come from different modalities. Visible light images capture reflected light and typically have high spatial resolution, considerable detail, and light-dark contrast, while infrared images capture thermal radiation and are resistant to some harsh environmental interference. This can provide greater information content, higher accuracy, and better robustness than single-modal signals. However, TIR-V image pairs are typically captured by different sensors, which can lead to potential resolution issues when combined. Moreover, both image types are 2D data, which lacks the ability to extract spatial information, making it difficult to detect and recognize targets in three-dimensional space. Depth and intensity (DI) fusion technology, which introduces distance information, can make up for the lack of depth information. Among various DI fusion schemes, RGB and depth image (RGB-D) fusion is currently one of the most widely used technologies. However, RGB-D image fusion requires the environment to provide good lighting conditions. In low light, unstable lighting conditions or completely dark environments, its acquisition effect and accuracy will be greatly reduced. Summary of the Invention

[0004] To address the problems existing in the prior art, the present invention provides a material recognition method and system. This method improves the accuracy of object recognition in different spatial locations and with different surface geometries. To achieve the above objectives, the present invention employs the following technical solutions:

[0005] In a first aspect, the present invention provides a material identification method, comprising:

[0006] Obtain the depth and active infrared reflection intensity data of the object to be identified and extract the object's posture information. The obtained data includes a depth map, an intensity map, and a point cloud map;

[0007] Perform contour extraction, pixel depth value and intensity value extraction on the depth map and intensity map respectively;

[0008] Based on the contour extraction, pixel depth value and intensity value extraction, the point cloud image is added to calculate the horizontal and vertical inclination angles related to each pixel point of the object to be identified;

[0009] The horizontal and vertical tilt angles, together with the extracted spatial position and depth values ​​of the pixel point, are input into the pre-trained optimized neural network model as the input data values ​​of the neural network. The fitting strength value of each pixel point is fitted, and the fitting result is output for the identification of the material of the object to be tested.

[0010] As a further improvement of the present invention, the training method of the pre-trained optimized neural network model includes:

[0011] Obtain the depth and active infrared reflection intensity data of the training object and extract the object's posture information. The obtained data includes depth map, intensity map and point cloud map;

[0012] The depth map and intensity map of the training object are respectively subjected to contour extraction, pixel depth value and intensity value extraction;

[0013] Based on the contour extraction, pixel depth value and intensity value extraction of the training object, the point cloud image of the training object is added to obtain the horizontal and vertical tilt angles related to each pixel point of the training object;

[0014] The horizontal and vertical tilt angles, together with the extracted spatial position and depth values ​​of the pixel points, are input into the pre-trained optimized neural network model as the input data values ​​of the neural network. The fitting strength value of each pixel point is fitted, and the fitting result is output for training the recognition of the object material. Finally, the optimized neural network model is trained.

[0015] As a further improvement of the present invention, the step of adding the point cloud image to calculate the horizontal tilt angle and vertical tilt angle associated with each pixel point of the object to be identified includes:

[0016] After acquiring the point cloud of the scene to be measured, irrelevant background information in the point cloud is filtered out, and only the point cloud information of the foreground object to be measured is retained; the point cloud of each object to be measured is extracted and saved separately in the foreground point cloud so as to obtain the normal vector information of each point;

[0017] After setting the parameters of the local surface model, local sphere neighborhood radius, and normal direction, and adjusting the direction of the point cloud normal using the minimum cost spanning tree, the normal vector value of each point in the image is calculated and saved, recorded as Nx, Ny, and Nz;

[0018] The direction of the line connecting the center of the field of view to the center of the camera lens is recorded as the z direction, and the direction parallel to the horizontal plane and perpendicular to the z axis is recorded as the y direction. A three-dimensional orthogonal coordinate system xyz is established in the field of view. The horizontal tilt angle α and vertical tilt angle β of each point in the point cloud are represented by the normal vectors Nx, Ny and Nz in three directions.

[0019]

[0020]

[0021] The horizontal tilt angle α and the vertical tilt angle β calculated for each point in the point cloud are averaged to obtain a quantitative description of the tilt of the object to be identified.

[0022] As a further improvement of the present invention, the fitting strength value of each pixel point is specifically:

[0023] The intensity value ir corresponding to each pixel of the material to be identified can be determined by six variables: the material to which it belongs, the depth value dep of the plane from the camera, its position in the plane (x, y), and the horizontal tilt angle α and vertical tilt angle β. Specifically:

[0024] ir=f(material,x,y,dep,α,β)

[0025] After fitting, we get

[0026] Pixel i corresponds to material ←g(x i ,y i ,dep i ,ir i,实际 ,ir i,拟合 )

[0027] Material of the object to be tested←Σ i g(x i ,y i ,dep i ,ir i,实际 ,ir i,拟合 )

[0028] Where x i is the horizontal position of pixel i, y i is the vertical position of pixel i. These two parameters are related to the resolution of the acquired image; dep i is the depth value of the plane where pixel i is located from the camera; α i is the horizontal inclination angle of the object at the position corresponding to pixel i; β i is the vertical inclination angle of the object at the position corresponding to pixel i; ir i,实际is the actual intensity value of the object at the position corresponding to pixel i; ir i,拟合 is the fitting intensity value of the object at the position corresponding to pixel i.

[0029] As a further improvement of the present invention, the step of outputting the fitting result for identifying the material of the object to be measured includes:

[0030] The fitting intensity value of each material is calculated by multiplying the fitting intensity value and the tilt factor one by one, and the blind spots are eliminated.

[0031] Calculate the intensity value deviation Δir i , and the relative deviation err i ;

[0032] For various materials Δir i , compare and get the minimum value Δir imin And its corresponding err imin , then Δir imin The corresponding material is regarded as the material corresponding to this pixel;

[0033] The material corresponding to each pixel is accumulated, and the material corresponding to the largest number of pixels is determined to be the material of the object to be tested.

[0034] As a further improvement of the present invention, the calculation of the fitting intensity value result of the pixel point corresponding to each material by multiplying the fitting intensity value and the tilt factor one by one, and eliminating the recognition blind spot, includes:

[0035] The specific quantitative criteria for eliminating blind spots are:

[0036] when

[0037]

[0038] and

[0039]

[0040] When , pixel j is considered to be a blind spot for judging material a and material b. Pixel j is first removed from the pixel points contained in the object to be tested. After all blind spots are removed, the method shown can be used to make a difference with the true intensity value to obtain the final material judgment result.

[0041] As a further improvement of the present invention, the material corresponding to each pixel is accumulated, and the material corresponding to the largest number of pixels is considered to be the material of the object to be measured, specifically including:

[0042] The tilt factor θ is introduced to describe the intensity change of each pixel due to the tilt condition. The tilt factor θ is defined as the ratio of the fitted intensity value of a pixel with horizontal tilt angle α and vertical tilt angle β to the fitted intensity value without tilt:

[0043]

[0044] According to the neural network assuming no tilt, when a certain pixel point is a certain material, its corresponding fitting intensity value ir is obtained. ia 、ir ib with ir ic Then, the tilt factor θ corresponding to the pixel point is obtained through the tilt neural network when it is a certain material. ia ,θ ib and θ ic , will ir ia 、ir ib with ir ic and θ ia ,θ ib and θ ic The results of multiplying them one by one are used as the fitting intensity value of the pixel corresponding to each material under the tilt condition, and then the intensity value deviation Δir is calculated. ia , Δir ib and Δir ic , and the relative deviation err ia 、err ib with err ic :

[0045] Δir ia =|ir ia ×θ ia -ir i |

[0046] Δir ib =|ir ib ×θ ib -ir i |

[0047] Δir ic =|ir ic ×θ ic -ir i |

[0048]

[0049]

[0050]

[0051] For Δir ia , Δirib and Δir ic Compare the three values ​​and get the minimum value Δir imin And its corresponding err imin , then Δir imin The corresponding material is regarded as the material corresponding to this pixel;

[0052] The material corresponding to each pixel is accumulated, and the material corresponding to the largest number of pixels is determined to be the material of the object to be tested.

[0053] In a second aspect, the present invention provides a material identification system, comprising:

[0054] The acquisition module is used to obtain the depth and active infrared reflection intensity data of the object to be identified and extract the object's posture information. The obtained data includes a depth map, an intensity map, and a point cloud map;

[0055] The extraction module is used to extract the contour, pixel depth value and intensity value of the depth map and intensity map respectively;

[0056] The calculation module is used to calculate the horizontal and vertical inclination angles of each pixel of the object to be identified based on the contour extraction, pixel depth value and intensity value extraction, and add the point cloud image;

[0057] The recognition module is used to input the horizontal inclination angle and vertical inclination angle, together with the extracted spatial position and depth value of the pixel point as the input data values ​​of the neural network into the pre-trained optimized neural network model, fit the fitting strength value of each pixel point, and output the fitting result for the identification of the material of the object to be tested.

[0058] In a third aspect, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the material identification method is implemented when the processor executes the computer program.

[0059] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the material identification method is implemented.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] The method of the present invention fully considers the influence of different plane positions of the object and different inclination angles of the surface normal vector in the collected space on the intensity value collected by the camera when the material with the same infrared reflection performance is used as the object to be identified, and comprehensively incorporates variables such as plane position and inclination into the identification method, thereby improving the accuracy of object identification in different spatial positions and different surface geometric conditions. The method extracts, processes and learns the grayscale value data of each pixel point of the collected material, avoiding the situation where some methods can only be used to identify single objects of a specific shape and size. The object recognition method accurate to the pixel level can meet the recognition requirements of objects of any shape composed of materials with the same infrared reflection performance within a certain range of sizes and a certain spatial position, provided that the resolution conditions permit. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Flowchart of the object material recognition method provided by the present invention;

[0063] Figure 2 It is a schematic diagram of the definition of the horizontal tilt angle α and the vertical tilt angle β introduced to jointly describe the tilt of a specific area on the surface of the material object to be identified; wherein, a is a schematic diagram of the definition of the horizontal tilt angle α in the front view perspective; b is a schematic diagram of the definition of the horizontal tilt angle α in the top view perspective; c is a schematic diagram of the definition of the vertical tilt angle β in the front view perspective; d is a schematic diagram of the definition of the vertical tilt angle β in the side view perspective;

[0064] Figure 3 This is a flow chart of the main data processing and utilization ideas of the method of the present invention;

[0065] Figure 4 It is a schematic diagram of a feasible method for material judgment strategy part of the present invention;

[0066] Figure 5 This is a diagram of the dataset collection scene and equipment of the present invention; wherein, 1-computer; 2-camera tripod; 3-ToF camera; 4-collection scene;

[0067] Figure 6 This is a schematic diagram of the present invention when collecting data sets;

[0068] Figure 7 This is a schematic diagram of a feasible situation of collecting data at different tilt angles in the same plane when the present invention collects data sets;

[0069] Figure 8Schematic diagram of the effects of the image data preprocessing and contour extraction process of the present invention, wherein (a) is a schematic diagram of the result of the depth map obtained after mean denoising (specifically, multiple images of the same scene are collected and averaged), (b) is a schematic diagram of the depth map result after depth value filtering, (c) is a schematic diagram of the depth map result after corrosion denoising, (d) is a schematic diagram of the depth map result after contour extraction, and (e) is a schematic diagram of the depth map result after contour binarization.

[0070] Figure 9 Schematic diagram of the object material recognition system provided by the present invention;

[0071] Figure 10 It is a schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0072] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0073] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0074] The fundamental purpose of this invention is to use depth cameras such as time-of-flight (ToF) cameras to obtain depth and active infrared reflection intensity (D-AI) data of objects with specific materials, and use the fused data and the obtained object posture information as the data basis for object material identification, thereby realizing material identification of objects of a certain size in different spatial positions and different surface geometries.

[0075] like Figure 1 As shown, the first purpose is to provide a material identification method, including:

[0076] S1, obtain the depth and active infrared reflection intensity data of the object to be identified and extract the object's posture information. The obtained data includes a depth map, an intensity map, and a point cloud map;

[0077] S2, extracting contours, pixel depth values ​​and intensity values ​​from the depth map and intensity map respectively;

[0078] S3, based on the contour extraction, pixel depth value and intensity value extraction, adds the point cloud image to calculate the horizontal tilt angle and vertical tilt angle related to each pixel point of the object to be identified;

[0079] S4, the horizontal tilt angle and the vertical tilt angle, together with the extracted spatial position and depth value of the pixel point, are input into the pre-trained optimized neural network model as the input data values ​​of the neural network, and the fitting strength value of each pixel point is fitted. The fitting result is output for the identification of the material of the object to be tested.

[0080] This method is based on the fusion technology of depth and active infrared intensity (D-AI), which can not only better adapt to the acquisition in dark environments, but also obtain three-dimensional stereo information of the target. The present invention uses a depth camera to simultaneously collect depth and active infrared intensity information, which will make it possible to achieve high-precision target detection using a single sensor.

[0081] Depth maps and point clouds directly captured by depth cameras often contain interference such as background and noise, requiring a series of image processing techniques for noise reduction and enhancement. Machine learning, a key research area in computer vision, has significant application value in object detection. Using machine learning algorithms, large sample sizes and high-accuracy object material recognition can be achieved.

[0082] This paper proposes a method for identifying object materials, capable of identifying the materials of objects of a certain size in different spatial locations and with different surface geometries. Specifically, this method utilizes depth cameras, such as time-of-flight (ToF) cameras, to acquire depth and active infrared reflection intensity (D-AI) data of objects made of specific materials, extracting the object's pose information, and using the fused data as the basis for object material identification to improve object recognition accuracy. This method, which falls within the fields of optical information processing and computer vision technology, performs object material identification.

[0083] As a specific example, the training method of the pre-trained optimized neural network model includes:

[0084] Obtain the depth and active infrared reflection intensity data of the training object and extract the object's posture information. The obtained data includes depth map, intensity map and point cloud map;

[0085] The depth map and intensity map of the training object are respectively subjected to contour extraction, pixel depth value and intensity value extraction;

[0086] Based on the contour extraction, pixel depth value and intensity value extraction of the training object, the point cloud image of the training object is added to obtain the horizontal and vertical tilt angles related to each pixel point of the training object;

[0087] The horizontal and vertical tilt angles, together with the extracted spatial position and depth values ​​of the pixel points, are input into the pre-trained optimized neural network model as the input data values ​​of the neural network. The fitting strength value of each pixel point is fitted, and the fitting result is output for training the recognition of the object material. Finally, the optimized neural network model is trained.

[0088] This method is designed based on the assumption that the intensity value ir corresponding to each pixel of an object to be identified can be determined by six variables: the material, the depth of the plane from the camera (dep), the position (x, y) within the plane, and the horizontal and vertical tilt angles α and β.

[0089] ir=f(material,x,y,dep,α,β)

[0090] This method introduces the horizontal tilt angle α and the vertical tilt angle β to describe the tilt of a specific area on the surface of the object to be identified. The horizontal tilt angle α is defined as the angle between the projection of the normal vector of the object surface on the horizontal plane and the line connecting the camera lens to the center of the field of view, as follows: Figure 2 As shown, the projection of the normal vector of the object surface is located to the right of the line connecting the camera lens to the center of the field of view, and the horizontal tilt angle is positive; otherwise, the horizontal tilt angle is negative. The plane passing through the line connecting the camera lens to the center of the field of view and perpendicular to the horizontal plane is defined as the vertical plane, and the vertical tilt angle β is defined as the angle between the projection of the normal vector of the object surface on the vertical plane and the line connecting the camera lens to the center of the field of view, as shown in Figure 2 As shown, the projection of the normal vector of the object surface is located on the upper side of the line connecting the camera lens and the center of the field of view, and the vertical tilt angle is positive; otherwise, the vertical tilt angle is negative.

[0091] The main idea of ​​this method is to collect and preprocess the depth intensity image dataset and the scene image to be collected, including noise reduction and enhancement methods such as mean denoising, depth value filtering, and corrosion denoising, as well as contour extraction, and extract pixel depth and intensity values. The method then adds the acquisition and preprocessing of the point cloud image of the scene to be tested to obtain the horizontal and vertical tilt angles associated with each pixel of the object to be identified. These are then used as the input data values ​​of the neural network together with the extracted spatial position and depth value of the pixel. Afterwards, an optimized neural network, including but not limited to GA-BP and DBO-BP, is used to train and fit the fitted intensity value of each pixel. Finally, the fitting results and the collected intensity values ​​are used to identify the material of the object to be tested using certain judgment methods and standards.

[0092] The specific processing methods involved in point cloud preprocessing and horizontal and vertical inclination angle extraction are as follows:

[0093] After acquiring a point cloud of the scene to be measured, the CloudCompare software is first used to filter out irrelevant background information from the point cloud, retaining only the point cloud information of the foreground object to be measured. Each point cloud of the object to be measured is then extracted and saved individually from the foreground point cloud to determine the normal vector information for each point. Using the Normals > Compute function in CloudCompare software, and by setting appropriate parameters such as the local surface model, local sphere neighborhood radius, and normal direction, and using the minimum spanning tree to adjust the orientation of the point cloud normal, the normal vector value for each point in the image can be calculated and saved, denoted as Nx, Ny, and Nz.

[0094] The z-direction is the line connecting the center of the field of view to the center of the camera lens, and the y-direction is the direction parallel to the horizontal plane and perpendicular to the z-axis. A three-dimensional orthogonal coordinate system xyz can be established in the field of view. In this coordinate system, the horizontal tilt angle α and vertical tilt angle β of each point in the point cloud can be expressed using the normal vectors Nx, Ny, and Nz in the three directions.

[0095]

[0096]

[0097] In this way, by taking the average value of the horizontal tilt angle α and the vertical tilt angle β calculated for each point in the point cloud image, a quantitative description of the tilt of the object to be identified can be obtained and applied to the neural network fitting.

[0098] For the application purpose of object material identification in this study, after the infrared reflection intensity data is trained and fitted by the neural network and the results are obtained, the above formula ir = f(material, x, y, dep, α, β) can be expressed as follows.

[0099] Pixel i corresponds to material ←g(x i ,y i ,dep i ,ir i,实际 ,ir i,拟合 )

[0100] Material of the object to be tested←Σ i g(x i ,y i ,dep i ,ir i,实际 ,ir i,拟 combine)

[0101] Where x i is the horizontal position of pixel i, y i is the vertical position of pixel i. These two parameters are related to the resolution of the acquired image; dep i is the depth value of the plane where pixel i is located from the camera; α i is the horizontal inclination angle of the object at the position corresponding to pixel i; β i is the vertical inclination angle of the object at the position corresponding to pixel i; ir i,实际 is the actual intensity value of the object at the position corresponding to pixel i; ir i,拟合 is the fitting intensity value of the object at the position corresponding to pixel i.

[0102] From this, it can be seen that as long as a fitting relationship that satisfies the formula ir=f(material,x,y,dep,α,β) is found, it is possible to determine the material of the object to be identified.

[0103] A feasible specific judgment method based on the above formula proposed according to the principle of the present invention is as follows, taking a scenario involving three materials (denoted as a, b, and c) as an example:

[0104] This method introduces a tilt factor θ to describe the intensity variation caused by the tilt of each pixel. The tilt factor θ can be defined as the ratio of the intensity value of the pixel with a horizontal tilt angle α and a vertical tilt angle β to the intensity value of the pixel without tilt, as shown in the following formula.

[0105]

[0106] The judgment method is to obtain the fitting intensity value ir corresponding to a certain material when a certain pixel point is based on the neural network assuming no tilt.ia 、ir ib with ir ic Then, the tilt factor θ corresponding to the pixel point is obtained through the tilt neural network when it is a certain material. ia ,θ ib and θ ic , will ir ia 、ir ib with ir ic and θ ia ,θ ib and θ ic The results of multiplying them one by one are used as the fitting intensity value of the pixel corresponding to each material under the condition of tilt, and then the intensity value deviation Δir is calculated. ia , Δir ib and Δir ic , and the relative deviation err ia 、err ib with err ic .

[0107] Δir ia =|ir ia ×θ ia -ir i |

[0108] Δir ib =|ir ib ×θ ib -ir i |

[0109] Δir ic =|ir ic ×θ ic -ir i |

[0110]

[0111]

[0112]

[0113] The strategy for judging the material, that is, for Δir ia , Δir ib and Δir ic Compare the three values ​​and get the minimum value Δir imin And its corresponding err imin , then Δir imin The corresponding material can be regarded as the material corresponding to the pixel point. The material corresponding to each pixel point is accumulated, and the material corresponding to the largest number of pixels is the material of the object to be tested determined by this method.

[0114] In the method of the present invention, at certain spatial locations, the intensity values ​​of the three materials corresponding to the pixel points respectively may partially overlap or be close, that is, at this location, there are two or more material intensity values ​​that are basically the same. For example, at the spatial location corresponding to a certain pixel point j,

[0115] ir ja ≈ir jb

[0116] This method records the pixels at these locations as blind spots. If the data corresponding to the blind spots are used for identification according to the method, it will be impossible to determine the specific material of these points, thereby affecting the overall object material judgment and causing a decrease in recognition accuracy. Therefore, before applying the method to calculate the difference and make a judgment, the data corresponding to the blind spots should first be eliminated according to certain standards. For the example shown, the specific quantitative standard selected by the present invention for eliminating blind spots is:

[0117] when

[0118]

[0119] and

[0120]

[0121] When , pixel j is considered to be a blind spot for judging material a and material b. Pixel j is first removed from the pixel points contained in the object to be tested. After all blind spots are removed, the method shown can be used to make a difference with the true intensity value to obtain the final material judgment result.

[0122] Figure 2 This is a schematic diagram of the definition of the horizontal tilt angle α and the vertical tilt angle β, which are respectively introduced in this method to jointly describe the tilt of a specific area on the surface of the material object to be identified. The horizontal tilt angle α is defined as the angle between the projection of the normal vector of the object surface on the horizontal plane and the line connecting the camera lens to the center of the field of view. Among them, a is a schematic diagram of the definition of the horizontal tilt angle α under the perspective of the front view; b is a schematic diagram of the definition of the horizontal tilt angle α under the perspective of the top view. If the projection of the normal vector of the object surface is to the right of the line connecting the camera lens to the center of the field of view, the horizontal tilt angle takes a positive value; otherwise, the horizontal tilt angle takes a negative value. The plane passing through the line connecting the camera lens to the center of the field of view and perpendicular to the horizontal plane is defined as a vertical plane, and the vertical tilt angle β is defined as the angle between the projection of the normal vector of the object surface on the vertical plane and the line connecting the camera lens to the center of the field of view. Among them, c is a schematic diagram of the definition of the vertical tilt angle β under the perspective of the front view; d is a schematic diagram of the definition of the vertical tilt angle β under the perspective of the side view. The projection of the normal vector on the object surface is located above the line connecting the camera lens and the center of the field of view, and the vertical tilt angle is positive; otherwise, the vertical tilt angle is negative.

[0123] Figure 3 This is a flowchart of the main data processing and utilization ideas of the method of the present invention. Specifically, on the basis of image acquisition, preprocessing (including noise reduction and enhancement methods such as mean denoising, depth value filtering, corrosion denoising, etc.) of the depth intensity image dataset and the image of the scene to be collected, contour extraction, pixel depth value and intensity value extraction, the acquisition and preprocessing process of the point cloud map of the scene to be tested are added to obtain the horizontal and vertical inclination angles related to each pixel point of the object to be identified, and the extracted spatial position and depth value of the pixel point are used as the input data value of the neural network. Afterwards, the optimized neural network including but not limited to GA-BP, DBO-BP, etc. is used for training and fitting the fitting intensity value of each pixel point. Finally, the fitting result is used for the identification of the material of the object to be tested using certain judgment methods and standards.

[0124] Figure 4 This is a schematic diagram of a feasible method for material judgment strategy of the present invention (taking three materials as an example). Specifically, the intensity value of the pixel corresponding to each material is calculated by fitting the intensity value and the tilt factor, and then the intensity value deviation Δir is calculated. ia , Δir ib and Δir ic , and the relative deviation err ia 、err ib with err ic , for Δir ia , Δir ib and Δir ic Compare the three values ​​and get the minimum value Δir imin And its corresponding err imin , then Δir imin The corresponding material can be regarded as the material corresponding to the pixel point. The material corresponding to each pixel point is accumulated, and the material corresponding to the largest number of pixels is the material of the object to be tested determined by this method.

[0125] Figure 5 This is a feasible diagram of the dataset collection scene and equipment of the present invention. The collection equipment mainly consists of a depth camera (TOF camera) 3, a computer 1, and a camera tripod 2. The object is the collection scene 4.

[0126] Among them, the material recognition method of the present invention is implemented by a computer, and the depth camera is mainly used to obtain the depth and active infrared reflection intensity (D-AI) data of objects with specific materials and extract the object posture information, and send the data to the computer.

[0127] Figure 6This illustrates a feasible method for collecting data from different locations within the same plane during dataset acquisition. The purpose of designing multiple locations is to cover the intensity distribution at as many plane locations as possible. This method can generate D-AI datasets for up to nine different plane locations at depths between 1.0 and 2.0 meters, effectively covering data at different depths and spatial locations, making the data more universal.

[0128] Figure 7 This figure illustrates a feasible scenario for collecting data at different tilt angles within the same plane during dataset acquisition. For intuitive presentation, the figure shows the result of contour extraction of the acquired depth map. This approach allows for the generation of D-AI datasets at various tilt angles, effectively capturing and demonstrating the impact and variation of different tilt angles on the material's active infrared reflectance intensity data, making the data more universal.

[0129] Figure 8 This is a flowchart of the present invention, illustrating the effects of the image data preprocessing and contour extraction process. (a) is a diagram illustrating the result of the depth map obtained after mean denoising (specifically, multiple images of the same scene are collected and averaged), (b) is a diagram illustrating the depth map result after depth value filtering, (c) is a diagram illustrating the depth map result after corrosion denoising, (d) is a diagram illustrating the depth map result after contour extraction, and (e) is a diagram illustrating the depth map result after contour binarization. Through this process, effective information within the target object range can be extracted, irrelevant information in the image can be eliminated, and the influence of noise and edge inaccuracy can be reduced as much as possible, so as to improve the effect of subsequent methods for object recognition.

[0130] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0131] Figure 3 This is an embodiment of the present invention. The processing steps are as follows:

[0132] Step 1: Perform image acquisition on the depth intensity image dataset and the scene image to be acquired respectively. The acquired data includes depth map, intensity map and point cloud.

[0133] Step 2: Preprocess the depth map, including noise reduction and enhancement methods such as mean denoising, depth value filtering, and corrosion denoising;

[0134] Step 3: Perform contour extraction, pixel depth value and intensity value extraction on the processed depth map and intensity map respectively; Gaussian filter contour extraction is used for contour extraction.

[0135] Step 4: Point cloud preprocessing process, by calculating the normal vector to obtain the horizontal and vertical inclination angles related to each pixel point of the object to be identified.

[0136] Step 5: The horizontal and vertical tilt angles, along with the extracted spatial position and depth of the pixel, are used as input data for the neural network. An optimized neural network, such as GA-BP and DBO-BP, is used to train and fit the strength value of each pixel. The fitting results are then used to identify the material of the object under test using a specific judgment method and standard.

[0137] Figure 4 The present invention relates to the above step 5, which uses the fitting results to adopt certain judgment methods and standards to perform material judgment.

[0138] Step 1: Calculate the fitting intensity value of each pixel corresponding to each material by multiplying the fitting intensity value and the tilt factor one by one, and eliminate the recognition blind spots;

[0139] Step 2: Calculate the intensity deviation Δir i , and the relative deviation err i .

[0140] Step 3: Δir for various materials i , compare and get the minimum value Δir imin And its corresponding err imin , then Δir imin The corresponding material can be regarded as the material corresponding to this pixel.

[0141] Step 4: Accumulate the material corresponding to each pixel point, and consider that the material corresponding to the largest number of pixels is the material of the object to be tested determined by this method.

[0142] Based on the above description, it can be concluded that the method of the present invention has at least the following advantages:

[0143] (1) Using the fusion of depth and active infrared reflection intensity for object recognition can not only be well applied to object recognition in three-dimensional scenes, but also avoid the adverse effects or interference caused by ambient lighting conditions. This allows object recognition in darkness or strong light conditions to still achieve the same accuracy as under normal lighting conditions, thus making it more applicable to application scenarios such as exploration, criminal investigation, security, and monitoring where good ambient lighting cannot be guaranteed.

[0144] (2) The fusion data of depth and active infrared reflection intensity is used as the image data basis for object recognition. The proposed method uses a single sensor for data acquisition, which can avoid the complex alignment process after the existing multiple sensors collect images to a certain extent.

[0145] (3) In the image preprocessing stage, the method continuously uses three effective noise reduction methods: mean denoising, depth value filtering, and corrosion denoising, so as to fully enhance the collected image data, filter out irrelevant depth information and noise, and minimize the multipath interference error, flying point noise error, and intensity error caused by the depth image during the acquisition process.

[0146] Compared with the patent CN202111054812.0, the focus of this application is to take into account the different plane positions and the inclination of the object surface under the same depth plane in the identification of the object material, that is, the influence of the object posture on the active infrared reflection intensity value of each pixel point of the object, thereby designing a method. However, the above patent does not take these two influencing factors into account. Therefore, the method proposed in the present invention can be better applied to the identification of objects of different sizes, different positions, and different inclination and curvature conditions on the surface within a certain range composed of the same material, thereby improving the accuracy of identification. In addition, the two methods have different processes and means for processing and extracting data. In addition, the present invention designs different material judgment strategies and standards.

[0147] like Figure 9 As shown, the third object of the embodiment of the present invention is to provide a material recognition system, comprising:

[0148] The acquisition module is used to obtain the depth and active infrared reflection intensity data of the object to be identified and extract the object's posture information. The obtained data includes a depth map, an intensity map, and a point cloud map;

[0149] The extraction module is used to extract the contour, pixel depth value and intensity value of the depth map and intensity map respectively;

[0150] The calculation module is used to calculate the horizontal and vertical inclination angles of each pixel of the object to be identified based on the contour extraction, pixel depth value and intensity value extraction, and add the point cloud image;

[0151] The recognition module is used to input the horizontal inclination angle and vertical inclination angle, together with the extracted spatial position and depth value of the pixel point as the input data values ​​of the neural network into the pre-trained optimized neural network model, fit the fitting strength value of each pixel point, and output the fitting result for the identification of the material of the object to be tested.

[0152] like Figure 10 As shown, the third purpose of an embodiment of the present invention is to provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the material identification method is implemented when the processor executes the computer program.

[0153] A fourth object of an embodiment of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the material identification method is implemented.

[0154] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be included in the scope of protection of the claims of the present invention. Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, or computer program products.

Claims

1. A material identification method, characterized in that: include: Obtain the depth and active infrared reflection intensity data of the object to be identified and extract the object's posture information. The obtained data includes a depth map, an intensity map, and a point cloud map; Perform contour extraction, pixel depth value and intensity value extraction on the depth map and intensity map respectively; Based on the contour extraction, pixel depth value and intensity value extraction, the point cloud image is added to calculate the horizontal and vertical inclination angles related to each pixel point of the object to be identified; The horizontal and vertical tilt angles, along with the extracted spatial position and depth values ​​of the pixel points, are input into a pre-trained optimized neural network model as input data values ​​of the neural network. The fitting strength value of each pixel point is fitted, and the fitting result is output for identifying the material of the object to be tested. The fitting intensity value of each pixel point is specifically: The intensity value corresponding to each pixel of the material object to be identified ir It can be determined by the material it belongs to and the depth value of the plane from the camera. dep , its position in the plane ( x , y ), and the horizontal inclination α and vertical inclination β These six variables are: After fitting, we get Where, x i Pixel i The horizontal position, y i Pixel i The vertical position, these two parameters are related to the resolution of the acquired image; dep i Pixel i The depth value of the plane from the camera; α i Pixel i The horizontal inclination angle of the object at the corresponding position; β i Pixel i The vertical inclination angle of the object at the corresponding position; ir i,实际 Pixel i The actual intensity value of the object at the corresponding position; ir i,拟合 Pixel i The fitting strength value of the object at the corresponding position; Outputting the fitting result for identifying the material of the object to be measured includes: The fitting intensity value of each material is calculated by multiplying the fitting intensity value and the tilt factor one by one, and the blind spots are eliminated. Calculate the intensity value deviation Δ ir i , and the relative deviation err i ; For various materials Δ ir i , compare and get the minimum value Δ ir imin and its corresponding err imin , then Δ ir imin The corresponding material is regarded as the material corresponding to this pixel; The material corresponding to each pixel is accumulated, and the material corresponding to the largest number of pixels is determined to be the material of the object to be tested.

2. A material identification method according to claim 1, characterized in that: The training method of the pre-trained optimized neural network model includes: Obtain the depth and active infrared reflection intensity data of the training object and extract the object's posture information. The obtained data includes depth map, intensity map and point cloud map; The depth map and intensity map of the training object are respectively subjected to contour extraction, pixel depth value and intensity value extraction; Based on the contour extraction, pixel depth value and intensity value extraction of the training object, the point cloud image of the training object is added to obtain the horizontal and vertical tilt angles related to each pixel point of the training object; The horizontal and vertical tilt angles, together with the extracted spatial position and depth values ​​of the pixel points, are input into the pre-trained optimized neural network model as the input data values ​​of the neural network. The fitting strength value of each pixel point is fitted, and the fitting result is output for training the recognition of the object material. Finally, the optimized neural network model is trained.

3. A material identification method according to claim 1, characterized in that: The step of adding the point cloud image to calculate the horizontal tilt angle and vertical tilt angle associated with each pixel point of the object to be identified includes: After acquiring the point cloud of the scene to be measured, irrelevant background information in the point cloud is filtered out, and only the point cloud information of the foreground object to be measured is retained; the point cloud of each object to be measured is extracted and saved separately in the foreground point cloud so as to obtain the normal vector information of each point; After setting the parameters of the local surface model, local sphere neighborhood radius, and normal direction, and adjusting the direction of the point cloud normal using the minimum cost spanning tree, the normal vector value of each point in the image is calculated and saved, which is recorded as Nx 、 Ny and Nz ; The direction of the line from the center of the field of view to the center of the camera lens is recorded as z direction, will be parallel to the horizontal plane and z The direction perpendicular to the axis is denoted as y Direction, establish a three-dimensional orthogonal coordinate system in the field of view xyz ; The horizontal inclination of each point in the point cloud α and vertical inclination β Using the normal vectors in three directions Nx 、 Ny and Nz To express; The horizontal inclination angle calculated for each point in the point cloud α and vertical inclination β , and the average value is used to obtain a quantitative description of the tilt of the object to be identified.

4. A material identification method according to claim 1, characterized in that: The calculation of the fitting intensity value result of the pixel corresponding to each material by multiplying the fitting intensity value and the tilt factor one by one and eliminating the recognition blind spots includes: The specific quantitative criteria for eliminating blind spots are: when and When the pixel To judge the material a and materials b The blind spot of the pixel j Eliminate the pixels contained in the object to be tested. After all blind spots are eliminated, the method shown can be used to make a difference with the true intensity value to obtain the final material judgment result.

5. A material identification method according to claim 1, characterized in that: The material corresponding to each pixel is accumulated, and the material corresponding to the largest number of pixels is determined as the material of the object to be tested. Specifically, the material includes: Introducing the tilt factor θ Describes the change in intensity caused by the tilt of each pixel; the tilt factor θ It is defined as a pixel point with a horizontal inclination α and vertical inclination β The ratio of the fitting strength value of to the fitting strength value without tilt: According to the neural network assuming no tilt, when a pixel is of a certain material, a 、 b 、 c Three materials and their corresponding fitting strength values ir ia 、 ir ib and ir ic Then, the tilt factor corresponding to the pixel point of a certain material is obtained through the tilt neural network. θ ia 、 θ ib and θ ic ,Will ir ia 、 ir ib and ir ic and θ ia 、 θ ib and θ ic The results of multiplying them one by one are used as the fitting intensity value of the pixel corresponding to each material under the condition of tilt, and then the intensity value deviation Δ is calculated. ir ia , Δ ir ib With Δ ir ic , and the relative deviation err ia 、 err ib and err ic : For Δ ir ia , Δ ir ib With Δ ir ic Compare the three values ​​and get the minimum value Δ ir imin and its corresponding err imin , then Δ ir imin The corresponding material is regarded as the material corresponding to this pixel; The material corresponding to each pixel is accumulated, and the material corresponding to the largest number of pixels is determined to be the material of the object to be tested.

6. A material identification system, characterized in that: include: The acquisition module is used to obtain the depth and active infrared reflection intensity data of the object to be identified and extract the object's posture information. The obtained data includes a depth map, an intensity map, and a point cloud map; The extraction module is used to extract the contour, pixel depth value and intensity value of the depth map and intensity map respectively; The calculation module is used to calculate the horizontal and vertical inclination angles of each pixel of the object to be identified based on the contour extraction, pixel depth value and intensity value extraction, and add the point cloud image; The recognition module is used to input the horizontal and vertical tilt angles, along with the extracted spatial position and depth value of the pixel point, as input data values ​​of the neural network into a pre-trained optimized neural network model, fit the fitting strength value of each pixel point, and output the fitting result for identifying the material of the object to be tested; The fitting intensity value of each pixel point is specifically: The intensity value corresponding to each pixel of the material object to be identified ir It can be determined by the material it belongs to and the depth value of the plane from the camera. dep , its position in the plane ( x , y ), and the horizontal inclination α and vertical inclination β These six variables are: After fitting, we get Where, x i Pixel i The horizontal position, y i Pixel i The vertical position, these two parameters are related to the resolution of the acquired image; dep i Pixel i The depth value of the plane from the camera; α i Pixel i The horizontal inclination angle of the object at the corresponding position; β i Pixel i The vertical inclination angle of the object at the corresponding position; ir i,实际 Pixel i The actual intensity value of the object at the corresponding position; ir i,拟合 Pixel i The fitting strength value of the object at the corresponding position; Outputting the fitting result for identifying the material of the object to be measured includes: The fitting intensity value of each material is calculated by multiplying the fitting intensity value and the tilt factor one by one, and the blind spots are eliminated. Calculate the intensity value deviation Δ ir i , and the relative deviation err i ; For various materials Δ ir i , compare and get the minimum value Δ ir imin and its corresponding err imin , then Δ ir imin The corresponding material is regarded as the material corresponding to this pixel; The material corresponding to each pixel is accumulated, and the material corresponding to the largest number of pixels is determined to be the material of the object to be tested.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the material identification method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the material identification method according to any one of claims 1 to 5.

Images