Surface identification method, device, equipment and product

Abstract

The invention discloses a surface recognition method, device, equipment and product, and relates to the field of image processing, and the method comprises the steps: projecting a preset coding image to a detected surface; acquiring a depth information coding image and an infrared bottom light coding image which are reflected by the measured surface; performing image matching according to the depth information coding image and a preset coding image to generate a depth map; performing texture feature extraction according to the infrared bottom light coding image to obtain a texture image; and according to the depth map and the texture map, performing prediction by using a multi-modal feature fusion classifier to obtain a prediction label of the surface. According to the invention, on the premise that the complexity of sensing hardware is not increased, accurate and multi-modal recognition can be carried out on the surface.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a surface recognition method, apparatus, device, and product. Background Technology

[0002] Existing ground recognition systems mostly rely on RGB images or TOF depth cameras, providing only a single dimension of information and struggling to accurately distinguish subtle differences between surfaces of different materials. In complex lighting environments, under obstructed or low-light conditions, the effectiveness of RGB images is significantly reduced. While structured light systems can provide high-precision 3D structures, they only cover the area illuminated by the structured light pattern, failing to form a complete texture image. Furthermore, they suffer from low reflectivity and poor signal-to-noise ratio on some material surfaces. Infrared imaging offers advantages such as resistance to light interference and privacy protection, but it lacks the ability to describe 3D structures.

[0003] Therefore, designing a method that can accurately and multimodally identify ground materials by fusing three-dimensional structural information with complete infrared texture information without increasing the complexity of sensing hardware is an important requirement in current technological development. Summary of the Invention

[0004] The purpose of this application is to provide a surface recognition method, apparatus, device, and product that can perform real-time, accurate, and multimodal recognition of surfaces without increasing the complexity of the sensing hardware.

[0005] To achieve the above objectives, this application provides the following solution:

[0006] In a first aspect, this application provides a surface identification method, comprising:

[0007] Acquire depth information encoded images and infrared bottom light encoded images after reflection from the tested surface;

[0008] Image matching is performed between the depth information encoded image and the preset encoded image to generate a depth map;

[0009] Texture features are extracted from the infrared background light encoded image to obtain a texture map;

[0010] Based on the depth map and texture map, a multimodal feature fusion classifier is used to predict the surface label.

[0011] In one embodiment, image matching is performed based on the depth information encoded image and a preset encoded image to generate a depth map, specifically including:

[0012] Image matching is performed based on the depth information encoded image and the preset encoded image to obtain a disparity map;

[0013] The depth value is obtained by converting the disparity map using the geometric parameters of the projection module and the imaging module.

[0014] A depth map is generated based on the depth value.

[0015] In one embodiment, texture feature extraction is performed based on the infrared background light encoded image to obtain a texture map, specifically including:

[0016] The infrared bottom light encoded image is subjected to image normalization and filtering preprocessing to obtain the processed infrared bottom light encoded image;

[0017] Based on the processed infrared bottom light encoded image, the local texture direction, local texture roughness, brightness variance, and reflection uniformity are determined respectively.

[0018] Construct a texture feature vector based on local texture direction, local texture roughness, brightness variance, and reflection uniformity;

[0019] The texture map is determined based on the texture feature vector.

[0020] In one embodiment, determining the texture map based on the texture feature vector specifically includes:

[0021] The scattering characteristics are determined based on the processed infrared bottom light encoded image;

[0022] Construct a texture feature enhancement vector based on the scattering features and the texture feature vector;

[0023] The texture map is determined based on the texture feature enhancement vector.

[0024] In one embodiment, a multimodal feature fusion classifier is used to predict the surface label based on the depth map and texture map, specifically including:

[0025] Feature extraction is performed on the depth map and the texture map respectively to obtain depth features and texture features;

[0026] The fusion features are determined based on the depth features and the texture features;

[0027] The predicted label of the surface is determined based on the fusion features.

[0028] In one embodiment, determining the fusion feature based on the depth feature and the texture feature specifically includes:

[0029] The fusion features are determined by splicing or attention weighting based on the depth features and the texture features.

[0030] In one embodiment, the training objective of the multimodal feature fusion classifier uses cross-entropy as the loss function.

[0031] Secondly, this application provides a surface identification device, comprising:

[0032] Projection module, imaging module, image processing module, and multimodal recognition module;

[0033] The projection module is used to output a preset encoded image;

[0034] The imaging module is used to acquire depth information encoded images and infrared bottom light encoded images after reflection from the surface being measured.

[0035] The image processing module is used to perform image matching based on the depth information encoded image and the preset encoded image to generate a depth map; the image processing module is also used to extract texture features based on the infrared background light encoded image to obtain a texture map;

[0036] The multimodal recognition module is used to make predictions based on the depth map and texture map using a multimodal feature fusion classifier to obtain the predicted label of the surface.

[0037] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the surface recognition method described above.

[0038] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the surface recognition method described above.

[0039] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0040] This application provides a surface recognition method, apparatus, device, and product. The method involves projecting a pre-coded image onto the surface to be measured; performing image matching between the depth information encoded image and the pre-coded image to generate a depth map; extracting texture features from the infrared bottom light encoded image to obtain a texture map; and using a multimodal feature fusion classifier based on the depth map and texture map to predict the surface's label. By acquiring high-precision depth information through the depth information encoded image and displaying the reflected image of the measured surface through the infrared bottom light encoded image, a single infrared camera can simultaneously acquire three-dimensional structure and texture images, enabling accurate, multimodal surface recognition. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is an application environment diagram of a surface recognition method according to an embodiment of this application;

[0043] Figure 2 A schematic flowchart of a surface recognition method provided in an embodiment of this application;

[0044] Figure 3 This is a schematic diagram of a conventional mesh structured light system.

[0045] Figure 4 A schematic diagram of grid structured light combined with infrared bottom light;

[0046] Figure 5 A functional module schematic diagram of a surface recognition device provided in another embodiment of this application;

[0047] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] The surface identification method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send the depth information encoded image and the infrared background light encoded image to be processed to server 104. After receiving the depth information encoded image and the infrared background light encoded image, server 104 performs image matching based on the depth information encoded image and a preset encoded image to generate a depth map; it extracts texture features based on the infrared background light encoded image to obtain a texture map; and it uses a multimodal feature fusion classifier to predict the surface based on the depth map and the texture map to obtain a predicted label. Server 104 can then feed back the obtained predicted surface label to terminal 102. In addition, in some embodiments, the surface recognition method can also be implemented by the server 104 or the terminal 102 alone. For example, the terminal 102 can directly perform surface recognition on the depth information encoded image and the infrared bottom light encoded image to be processed, or the server 104 can obtain the depth information encoded image and the infrared bottom light encoded image to be processed from the data storage system and perform surface recognition on the image to be processed.

[0051] The terminal 102 can be, but is not limited to, various robots, desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0052] In one exemplary embodiment, such as Figure 2 As shown, a surface recognition method is provided. This method is executed by a computer device, specifically, it can be executed by a computer device such as a terminal or a server alone, or it can be executed by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the following steps are included.

[0053] Step 201: Project a preset coded image onto the surface to be tested.

[0054] Step 202: Obtain the depth information encoded image and the infrared bottom light encoded image after reflection from the surface being measured.

[0055] Step 203: Perform image matching based on the depth information encoded image and the preset encoded image to generate a depth map.

[0056] Step 204: Extract texture features from the infrared background light encoded image to obtain a texture map.

[0057] Step 205: Based on the depth map and texture map, a multimodal feature fusion classifier is used to predict the surface label.

[0058] By implementing steps 201 to 205 above, high-precision depth information of the measured surface is obtained through depth information encoded images, and texture feature information of the measured surface is extracted through infrared bottom light encoded images. Thus, the simultaneous acquisition of three-dimensional structure and texture images can be achieved using a single light source and a single camera, thereby completing accurate, multimodal recognition of the measured surface.

[0059] In an exemplary embodiment, image matching is performed based on the depth information encoded image and the preset encoded image to generate a depth map. Specifically, this includes: performing image matching based on the depth information encoded image and the preset encoded image to obtain a disparity map; converting the disparity map using the geometric parameters of the projection module and the imaging module to obtain a depth value; and generating a depth map based on the depth value.

[0060] In practical applications, a preset coded image I is projected onto the surface being measured via a projection module. p (x p y p Patterns such as dot matrix, stripes, and grids implicitly contain spatial information such as phase, frequency, and position in their encoding. This pattern I... p (x p y p After being reflected and deformed by the surface being measured, the depth information is encoded into an image I by the imaging module. c (x c y c Due to differences in surface depth, the depth-encoded image will experience a spatial offset (Δx, Δy) relative to the projected preset-encoded image. Here, Δx is the horizontal offset between the projected preset-encoded image and the depth-encoded image, i.e., horizontal parallax; Δy is the vertical offset between the projected preset-encoded image and the depth-encoded image, i.e., vertical parallax. The imaging module acquires images, and the same image information includes both depth information and infrared background light information.

[0061] Pattern matching refers to the process of calculating the three-dimensional topography (i.e., depth) of a measured surface by establishing a pixel-level correspondence between a pre-coded image projected by a projection module and a depth information coded image acquired by an imaging module, combined with the geometric parameters of the projection and imaging modules. All pattern matching methods are existing technologies, including but not limited to:

[0062] Methods such as grayscale stripe coding + phase coding, dot matrix center detection + nearest neighbor matching, and feature descriptor (such as SIFT) matching are used. The matching result is an output disparity map, which generates a two-dimensional matrix representing the spatial offset (Δx, Δy) of each pixel by establishing a pixel-level correspondence between the preset coded image and the depth information coded image.

[0063] Using the geometric parameters of the projection and imaging modules, the horizontal parallax Δx is converted into a depth value Z:

[0064]

[0065] Where B is the horizontal baseline distance between the projection module and the imaging module; f is the focal length of the imaging module. Generating a depth map refers to normalizing the depth values ​​and then converting them into a depth map, specifically:

[0066] Normalization: Since the range of depth values ​​can be very large, directly using them as pixel values ​​can cause image display problems. Therefore, it is necessary to normalize the depth values, mapping them to a grayscale range of 0-255, to obtain the normalized depth value D(x, y).

[0067] Depth map generation: The normalized depth values ​​D(x, y) are mapped to a grayscale image to generate a depth map D. The depth map is typically a single-channel image (grayscale image), where each pixel value represents the depth of that point (distance from the imaging module). Pixel values ​​are proportional to depth; within the normalization range, the greater the depth, the larger the pixel value.

[0068] In an exemplary embodiment, texture feature extraction is performed on the infrared backlight coded image to obtain a texture map. Specifically, this includes image normalization and filtering preprocessing of the infrared backlight coded image to obtain a processed infrared backlight coded image; determining local texture direction, local texture roughness, brightness variance, and reflection uniformity based on the processed infrared backlight coded image; constructing a texture feature vector based on the local texture direction, local texture roughness, brightness variance, and reflection uniformity; and determining the texture map based on the texture feature vector. The reflection uniformity is determined based on the brightness variance.

[0069] In practical applications, infrared backlight illuminates the surface being measured, and after reflection, the imaging module acquires an infrared backlight encoded image I. IR After image normalization and filtering preprocessing, texture directionality extraction, local texture roughness, brightness variance, and reflection uniformity calculation, a texture feature vector F is constructed. texture Finally, according to F texture The texture map T is obtained through reasoning.

[0070] Main processing steps:

[0071] (1) Image normalization and filtering preprocessing.

[0072] The image is mean-filtered and normalized to eliminate brightness deviations:

[0073]

[0074] Among them, I norm (x, y) represents the processed infrared backlight encoded image, I IR (x, y) represents the infrared bottom light encoded image, μ is the global mean of the image, and σ is the global standard deviation of the image.

[0075] (2) Extraction of local texture direction.

[0076] Local texture orientation can be extracted using Gabor filter banks or Sobel operators.

[0077] T θ (x,y)=I norm (x,y)×G θ (x,y).

[0078] T θ (x,y) is the local texture direction, G θ (x,y) is the Gabor kernel at the relative angle θ between the projection module and the imaging module, which can cover multiple directions, such as 0°, 45°, 90°, and 135°.

[0079] (3) Calculation of local texture roughness.

[0080] Using local variance as a measure of local texture roughness:

[0081]

[0082] Among them, R local (x,y) represents the local texture roughness, N is a local window, and I norm (i,j) is the processed infrared bottom light encoded image I. norm (x, y) represents the grayscale value of any pixel within a local window. It is the average gray value within a local window.

[0083] (4) Luminance variance V lum calculate.

[0084]

[0085] Where M and N' are the height and width of the image, respectively, and μ' is the average brightness value of the image.

[0086] (5) Reflection uniformity Uref calculate.

[0087]

[0088] Reflection uniformity is used to determine information such as color and material changes.

[0089] (6) Construct texture feature vector F texture .

[0090] F texture =[T θ (x,y),R local (x,y), V lum U ref ].

[0091] In one exemplary embodiment, determining the texture map based on the texture feature vector specifically includes:

[0092] The scattering features are determined based on the processed infrared bottom light encoded image; a texture feature enhancement vector is constructed based on the scattering features and the texture feature vector; and a texture map is determined based on the texture feature enhancement vector.

[0093] In practical applications, further estimation of scattering characteristics (such as subsurface scattering attenuation coefficient μ”, surface roughness δ, anisotropy parameter α) is required. x α y The specular / diffuse reflection ratio ρ is used to construct the enhanced texture feature vector F'. texture This allows for more accurate analysis of the shape and intensity of bright spots in images, the directional distribution of reflection intensity, and edge blurring, providing richer information about material surfaces and enabling further differentiation between soft, hard, and hybrid materials.

[0094] Estimate the specular / diffuse reflection ratio:

[0095] The proportion ρ can be estimated by the weights of the diffuse and specular reflection terms in the BRDF model (the expression is complex and not shown), or a simplified expression can be used:

[0096]

[0097] Where ρ is the proportion of specular reflection to diffuse reflection, a fundamental parameter for distinguishing material types, reflecting the overall tendency of the material's surface reflection characteristics. W speular and W diffuse These represent the weights of specular reflection and diffuse reflection, respectively.

[0098] Estimate the subsurface scattering attenuation coefficient μ”:

[0099] μ quantifies the degree of light scattering within a material, affecting its transparency and edge blurring.

[0100] The subsurface scattering attenuation coefficient μ is retrieved based on the diffusion approximation model.

[0101]

[0102] Among them, L subsurface d represents the intensity of light emitted from the measured surface after subsurface scattering; d represents the depth to which the light penetrates the material.

[0103] Estimate surface roughness δ:

[0104] δ is a statistical parameter representing the microscopic geometric properties of the material surface, which affects the anisotropy and edge blurring of diffuse reflection. In the BRDF model, it is usually represented by the normal distribution function, commonly the Beckmann distribution or the GGX distribution.

[0105] Estimate the anisotropy parameter a x ,a y :

[0106] a x ,a y These describe the roughness of the specular reflection in two orthogonal directions, which affect the shape of the bright spot and the directional distribution of the reflection intensity. They can be obtained through inversion using the anisotropic Ward model or the anisotropic Beckmann model.

[0107] Obtain the texture feature enhancement vector F' texture =[T θ (x,y),R local (x,y), V lum U ref ,μ,μ”,δ,a x a y ].

[0108] (7) Based on the texture feature vector F texture Or texture feature enhancement vector F' texture The texture map T is obtained through reasoning.

[0109] Normalize each feature dimension of the texture feature vector, select an appropriate mapping method (such as direct mapping, RGB mapping, or fusion mapping), and generate a texture map T(x, y), where each pixel value or color channel comprehensively reflects the texture features of that point. Post-processing such as smoothing and contrast enhancement is then applied to the texture map to improve visualization.

[0110] In an exemplary embodiment, predicting a surface label using a multimodal feature fusion classifier based on the depth map and texture map specifically includes: extracting features from the depth map and texture map respectively to obtain depth features and texture features; determining a fusion feature based on the depth features and texture features; and determining the surface label based on the fusion feature. Specifically, determining the fusion feature based on the depth features and texture features includes: determining the fusion feature using a concatenation or attention-weighted approach based on the depth features and texture features.

[0111] In practical applications, this system uses a multi-branch convolutional neural network as the classifier. Multi-branch means that the network front end has two or more input paths (branches), each receiving different types of data or modalities (e.g., depth maps, texture maps), extracting features from each, and finally fusing and judging them in the middle and later stages. The core steps are as follows:

[0112] 1. Obtain the depth map D and texture map T from the image processing module.

[0113] 2. Extract the depth features F from the depth map D respectively. D Texture features F of texture map T T .

[0114] 3. Fuse deep features F through splicing or attention weighting. D Texture features F of texture map T T Obtain the fusion feature F fusion :

[0115] splicing:

[0116] F fusion =Concat(F D ,F T ).

[0117] In programming and data processing, concat typically refers to the operation of joining or merging two or more sequences, such as strings, arrays, or data frames.

[0118] Attention-weighted fusion:

[0119]

[0120] Where α is the weight coefficient, representing the depth feature F D The importance of.

[0121] 4. Discrimination Module:

[0122] The fused features are processed through a fully connected layer and a Softmax classifier to output the predicted label.

[0123] The training objective uses the cross-entropy loss function:

[0124]

[0125] Where L is the cross-entropy loss function; y i For the true distribution, This represents the distribution predicted by the model. The smaller the cross-entropy value, the closer the two distributions are.

[0126] It supports transfer learning and online fine-tuning.

[0127] Fully connected layers are the most basic and widely used layer type in neural networks. Their main functions include:

[0128] Feature integration: Integrating the features extracted from the previous layers into the final representation for classification, regression or other tasks.

[0129] Nonlinear mapping: Nonlinear activation functions, such as ReLU, sigmoid, or tanh, are typically applied after feature integration to increase the expressive power of the network.

[0130] Parameter learning: Fully connected layers contain a large number of learnable parameters, which are adjusted during training through backpropagation and gradient descent, enabling the network to adapt to the training data and make accurate predictions.

[0131] Output prediction: In classification tasks, the output of a fully connected layer is used to predict the probability distribution of the input belonging to each class. The softmax function is typically used to convert the score into class probabilities. Fully connected layers are widely used in tasks such as image classification, text processing, and regression analysis.

[0132] The Softmax classifier is a commonly used classifier in machine learning and deep learning, particularly suitable for multi-class classification problems. Its main function is to transform the input vector into a probability distribution, so that each class has a corresponding probability value, and the sum of the probabilities of all classes is 1.

[0133] Cross entropy is an important concept in information theory, used to measure the distance between one probability distribution and another. In machine learning, cross entropy is often used to evaluate the difference between the probability distribution predicted by the model and the true distribution.

[0134] In one exemplary embodiment, the training objective of the multimodal feature fusion classifier employs the cross-entropy loss function.

[0135] The method provided in this application takes ground material recognition as an example. This method is also applicable to the recognition of spots, moles, pimples, bumps, blood vessels, etc. on the skin surface. Specific application scenarios include: ground type perception and path adaptation for intelligent devices such as home service robots and cleaning robots; detection of road paving materials and boundaries in intelligent vehicle systems; ground quality, wear, and material detection in industrial or construction fields; privacy-friendly environmental perception in intelligent security and indoor navigation systems; and skin surface detection (medical aesthetics field).

[0136] This application relates to computer vision, infrared imaging, optical shaping, and artificial intelligence recognition technologies, providing a surface recognition method that combines structured light depth information and infrared texture information for multimodal fusion. It is applicable to applications such as robot navigation, path planning, intelligent cleaning and ground condition monitoring, skin surface and other surface detection. The structured light and infrared bottom light fusion illumination scheme, combined with single-camera multimodal acquisition and an AI classifier, acquires high-precision depth information through structured light encoded patterns, while simultaneously generating a complete ground reflection image using infrared diffuse bottom light. This allows for the simultaneous acquisition of 3D structure and texture images through a single light source and a single camera, enabling real-time, low-cost, and reliable surface perception. It can fuse 3D structural information and complete infrared texture information without increasing the complexity of the sensing hardware, thereby achieving accurate, multimodal surface recognition.

[0137] Based on the same inventive concept, this application also provides a surface recognition device for implementing the surface recognition method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more surface recognition device embodiments provided below can be found in the limitations of the surface recognition method described above, and will not be repeated here.

[0138] In one exemplary embodiment, such as Figure 5 As shown, a surface recognition device is provided for use in the surface recognition method described above. The surface recognition device includes a projection module, an imaging module, an image processing module, and a multimodal recognition module.

[0139] The projection module is used to output a preset encoded image.

[0140] The imaging module is used to acquire depth information encoded images and infrared bottom light encoded images after reflection from the surface being measured.

[0141] The image processing module is used to perform image matching based on the depth information encoded image and the preset encoded image to generate a depth map; the image processing module is also used to extract texture features based on the infrared background light encoded image to obtain a texture map.

[0142] The multimodal recognition module is used to make predictions based on the depth map and texture map using a multimodal feature fusion classifier to obtain the predicted label of the surface.

[0143] The various modules of the device are connected via electrical control. Structured light involves structuring light, including point structured light, line structured light, area structured light, and optical pattern encoding. After the structured light is projected onto the surface being measured through the projection module, the depth of the surface is modulated. The modulated structured light is acquired by the imaging module and transmitted to the image processing module for analysis and calculation, yielding the three-dimensional surface shape data of the measured surface. Modulation methods can be broadly categorized into temporal modulation and spatial modulation. The most commonly used temporal modulation method is the time-of-flight method. Spatial modulation methods involve changes in the phase, intensity, and other properties of the structured light field after depth modulation by the measured surface; by reading these changes, the surface shape information of the measured surface can be obtained. This system employs optical pattern encoded structured light and spatial modulation methods.

[0144] Projection module: Utilizing a single infrared laser (VCSEL or EEL, etc.), paired with a specific optical system or component (such as DOE diffractive optical elements or dynamic projection devices like LCD, DMD, MEMS, LCOS, etc.), it simultaneously outputs structured light encoded patterns (such as dot matrix, stripes, grids, etc.) within the same beam, generating a relatively uniformly diffused bottom light in the structured light region, achieving single-source composite illumination output. For example... Figure 3 and Figure 4 As shown, where, Figure 3 This is a schematic diagram of a conventional mesh structured light; Figure 4 This is a schematic diagram of a grid structured light combined with infrared bottom light.

[0145] Imaging module: Equipped with a camera and integrated narrowband filter, this module matches the wavelength of the light source used to acquire reflected images under fused illumination, ensuring system compactness. The camera supports global shutter acquisition and external triggering, and achieves precise synchronization with the light source module via a controller.

[0146] Image processing module: Simultaneously extracts structured light deformation information (for calculating depth maps) and infrared diffuse reflection information (for texture analysis) from the image.

[0147] Multimodal recognition module: Combines depth map and texture map features into a multimodal feature fusion classification model for fusion judgment, thereby realizing the identification and classification of the type of material on the probed surface.

[0148] The effects of this application are as follows:

[0149] 1. Enables simultaneous acquisition of infrared structured light and infrared images, eliminating the need for multiple imaging modules and reducing system cost and complexity.

[0150] 2. The system has a compact structure and can be integrated into a size-constrained embedded platform.

[0151] 3. It is adaptable to different ambient lighting conditions and can operate stably under conditions such as low light, strong light, and nighttime.

[0152] 4. Multimodal fusion significantly improves recognition performance on surfaces with weak texture and low contrast.

[0153] 5. Using the infrared band avoids collecting visible image content, which helps protect user privacy and is suitable for home, office and other scenarios.

[0154] 6. Simultaneously perceive the 3D morphology of the surface and the infrared reflection texture to achieve more discriminative material recognition.

[0155] 7. The system can perform online learning and model iteration, and supports continuous adaptation to different materials.

[0156] 8. The modular design facilitates expansion and can be adapted to scenarios such as wall recognition and surface quality inspection.

[0157] 9. Low power consumption design, suitable for mobile devices and long-running devices.

[0158] 10. Provide more refined ground semantic information support for robot path planning and cleaning strategy formulation.

[0159] In summary, this application features a compact structure, controllable cost, and high recognition accuracy, overcoming the shortcomings of existing RGB or single depth sensing schemes. It is particularly suitable for embedded intelligent systems in ground environment understanding and has significant engineering and commercial value.

[0160] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 6 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores surface identification data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a surface identification method.

[0161] Those skilled in the art will understand that Figure 6 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method embodiments.

[0162] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method embodiments.

[0163] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method embodiments.

[0164] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0165] In this application, all actions to acquire signals, information, or data are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with the authorization granted by the owner of the relevant device.

[0166] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0167] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0168] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A surface recognition method, characterized in that, The surface identification method includes: Project a pre-coded image onto the surface being measured; Acquire depth information encoded images and infrared bottom light encoded images after reflection from the tested surface; Image matching is performed between the depth information encoded image and the preset encoded image to generate a depth map; Texture features are extracted from the infrared background light encoded image to obtain a texture map; Based on the depth map and texture map, a multimodal feature fusion classifier is used to predict the surface label.

2. The surface recognition method according to claim 1, characterized in that, Image matching is performed between the depth information encoded image and a preset encoded image to generate a depth map, specifically including: Image matching is performed based on the depth information encoded image and the preset encoded image to obtain a disparity map; The depth value is obtained by converting the disparity map using the geometric parameters of the projection module and the imaging module. A depth map is generated based on the depth value.

3. The surface recognition method according to claim 1, characterized in that, Texture features are extracted from the infrared background light encoded image to obtain a texture map, specifically including: The infrared bottom light encoded image is subjected to image normalization and filtering preprocessing to obtain the processed infrared bottom light encoded image; Based on the processed infrared bottom light encoded image, the local texture direction, local texture roughness, brightness variance, and reflection uniformity are determined respectively. Construct a texture feature vector based on local texture direction, local texture roughness, brightness variance, and reflection uniformity; The texture map is determined based on the texture feature vector.

4. The surface identification method according to claim 3, characterized in that, Determining the texture map based on the texture feature vector specifically includes: The scattering characteristics are determined based on the processed infrared bottom light encoded image; Construct a texture feature enhancement vector based on the scattering features and the texture feature vector; The texture map is determined based on the texture feature enhancement vector.

5. The surface identification method according to claim 1, characterized in that, Based on the depth map and texture map, a multimodal feature fusion classifier is used to predict the surface label, specifically including: Feature extraction is performed on the depth map and the texture map respectively to obtain depth features and texture features; The fusion features are determined based on the depth features and the texture features; The predicted label of the surface is determined based on the fusion features.

6. The surface identification method according to claim 5, characterized in that, The fusion features are determined based on the depth features and the texture features, specifically including: The fusion features are determined by splicing or attention weighting based on the depth features and the texture features.

7. The surface recognition method according to claim 1, characterized in that, The training objective of the multimodal feature fusion classifier is to use the cross-entropy loss function.

8. A surface identification device, characterized in that, The surface recognition device is used to implement the surface recognition method according to any one of claims 1-7, and the surface recognition device includes: a projection module, an imaging module, an image processing module, and a multimodal recognition module; The projection module is used to output a preset encoded image; The imaging module is used to acquire depth information encoded images and infrared bottom light encoded images after reflection from the surface being measured. The image processing module is used to perform image matching based on the depth information encoded image and the preset encoded image to generate a depth map; the image processing module is also used to extract texture features based on the infrared background light encoded image to obtain a texture map; The multimodal recognition module is used to make predictions based on the depth map and texture map using a multimodal feature fusion classifier to obtain the predicted label of the surface.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the surface recognition method according to any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the surface recognition method according to any one of claims 1-7.

Images