Data processing method, device and equipment applied to material identification

Abstract

Embodiments of the present application provide a data processing method applied to material identification, which can be applied to the fields of artificial intelligence technology and robot technology. The data processing method applied to material identification comprises: determining at least one candidate surface region in a candidate surface region set as an association target region according to a candidate association score; and determining pixel window data of the association target region in an image coordinate system based on an axial effective depth of a vision sensor coordinate system corresponding to an initial ranging distance of a ranging measurement point in a pre-calibrated ranging sensor coordinate system, and in combination with a pre-calibration parameter, the pixel window data being used to realize material identification. Embodiments of the present application further provide a data processing apparatus, device, storage medium and program product applied to material identification.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to the interdisciplinary field of robot vision technology and image processing technology, and more specifically to a data processing method, apparatus, equipment, medium, and product applied to material identification. Background Technology

[0002] In the processing and sorting of wood, metal, plastic, rubber, foam, composite boards, and surface coatings, material category directly affects cutting parameters, spindle speed, feed rate, tool selection, and safety strategies. Desktop CNC (Computer Numerical Control), smart warehousing, and flexible production lines typically aim to automatically determine workpiece material before processing. Manual observation and experience-based judgment have poor consistency; near-infrared, hyperspectral, or X-ray equipment is costly; and relying solely on ordinary RGB images can easily lead to misidentification due to similar colors, coating coverage, changes in distance, and variations in lighting. Summary of the Invention

[0003] In view of at least one of the above problems, embodiments of the present invention aim to provide a data processing method, apparatus, device, medium, and product applicable to material identification.

[0004] One aspect of the present invention provides a data processing method for material identification, comprising: extracting a set of candidate surface regions for each frame of a received monocular visual image of a target to be identified; determining a candidate association score for each candidate surface region in the set of candidate surface regions based on a candidate set of projection points of the ranging measurement point of the target to be identified in the image coordinate system corresponding to the monocular visual image; determining at least one candidate surface region in the set of candidate surface regions as an associated target region based on the candidate association score; and determining pixel window data of the associated target region in the image coordinate system based on the axial effective depth of the visual sensor coordinate system corresponding to the initial ranging distance of the ranging measurement point in the pre-calibrated ranging sensor coordinate system, and in combination with pre-calibration parameters, wherein the pixel window data is used to realize material identification.

[0005] According to an embodiment of the present invention, before extracting a set of candidate surface regions for each frame of monocular visual image of the received target to be identified, the method further includes: acquiring each frame of original monocular image of the target to be identified through the visual sensor based on the joint calibration relationship between the visual sensor and the ranging sensor, and simultaneously acquiring the initial ranging distance of the target to be identified through the ranging sensor based on the ranging measurement point; performing image preprocessing on each frame of original monocular image to generate a monocular visual image; and performing depth processing on the initial ranging distance based on the joint calibration relationship to generate the original axial depth of the corresponding ranging measurement point in the coordinate system of the visual sensor.

[0006] According to an embodiment of the present invention, after performing image preprocessing on each frame of the original monocular image to generate a monocular visual image, and performing depth processing on the initial ranging distance based on the joint calibration relationship to generate the original axial depth of the corresponding ranging measurement point in the visual sensor coordinate system, the method further includes: determining the projection coordinate point of the ranging measurement point in the visual sensor coordinate system based on the pixel projection point of the monocular visual image corresponding to the joint calibration relationship in the visual sensor coordinate system; and determining the projection pixel coordinate of the ranging measurement point in the corresponding monocular visual image based on the projection coordinate point and the visual sensor intrinsic parameters of the pre-calibrated parameters.

[0007] According to an embodiment of the present invention, after performing image preprocessing on each frame of the original monocular image to generate a monocular visual image, and performing depth processing on the initial ranging distance based on the joint calibration relationship to generate the original axial depth of the corresponding ranging measurement point in the visual sensor coordinate system, the method further includes: performing median filtering on the original axial depth to determine the filtered axial depth of the corresponding ranging measurement point; performing exponential smoothing on the filtered axial depth to determine the smoothed axial depth of the corresponding ranging measurement point; determining the validity of the smoothed axial depth of the ranging measurement point corresponding to the monocular visual image, and performing mean processing on at least one historical valid depth and the smoothed axial depth determined by the validity determination to determine the axial valid depth in the visual sensor coordinate system.

[0008] According to an embodiment of the present invention, in extracting a candidate surface region set for each frame of a received monocular visual image of a target to be identified, the method includes: performing detection and segmentation on the monocular visual image to extract multiple candidate surface regions as a candidate surface region set.

[0009] According to an embodiment of the present invention, in determining the candidate association score corresponding to each candidate surface region in the candidate surface region set based on the candidate set of projection points of the ranging measurement points for the target to be identified in the image coordinate system corresponding to the monocular vision image, the method includes: determining the candidate association score corresponding to each candidate surface region based on the candidate set of projection points corresponding to the candidate surface regions containing the projection pixel coordinates, combined with a preset normalization parameter.

[0010] According to an embodiment of the present invention, determining at least one candidate surface region in a set of candidate surface regions as a target region for association based on the candidate association score includes: sorting each candidate surface region in the set of candidate surface regions according to the value of the candidate association score of each candidate surface region to generate a set of candidate regions; and selecting the candidate surface region with the largest candidate association score in the set of candidate regions as the target region for association.

[0011] According to an embodiment of the present invention, the determination of the pixel window data of the associated target region in the image coordinate system based on the axial effective depth of the initial ranging distance of the ranging measurement point in the pre-calibrated ranging sensor coordinate system and in combination with the pre-calibration parameters includes: determining the pixel size information of the preset actual texture window in the image coordinate system based on the physical size information of the preset actual texture window corresponding to the associated target region, the pixel focal length parameter of the pre-calibration parameters, and the axial effective depth.

[0012] Another aspect of the present invention provides a data processing apparatus for material identification, comprising a region extraction module, a score determination module, a region determination module, and a window determination module. The region extraction module extracts a set of candidate surface regions for each frame of a received monocular visual image of a target to be identified. The score determination module determines a candidate association score for each candidate surface region in the candidate surface region set based on a candidate set of projection points of the ranging point of the target to be identified in the image coordinate system corresponding to the monocular visual image. The region determination module determines at least one candidate surface region in the candidate surface region set as an associated target region based on the candidate association score. The window determination module determines pixel window data of the associated target region in the image coordinate system based on the axial effective depth of the visual sensor coordinate system corresponding to the initial ranging distance of the ranging point in the pre-calibrated ranging sensor coordinate system, and in conjunction with pre-calibration parameters. The pixel window data is used to achieve material identification.

[0013] Another aspect of the present invention provides an electronic device including one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the data processing method described above for material identification.

[0014] Another aspect of the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the data processing method described above for material identification.

[0015] Another aspect of the present invention provides a computer program product including a computer program that, when executed by a processor, implements the above-described data processing method for material identification.

[0016] The data processing method for material identification provided in this invention can at least partially solve at least one of the problems in related technologies, such as the inconsistency in the actual physical size observed by the network due to changes in texture pixel scale at different detection distances based on monocular vision detection, and poor stability of consecutive frames, which together cause poor material identification accuracy and efficiency. Therefore, it can achieve at least one of the following technical effects:

[0017] (1) First, the camera axial depth obtained by single-point ranging (such as TOF) is used to normalize the image texture block to a fixed actual millimeter scale, thereby achieving texture normalization according to physical scale and significantly improving cross-distance recognition consistency.

[0018] (2) Secondly, the image projection point of the measurement ray of single-point ranging is associated with the visual candidate region (such as ToF spot-ROI association) to ensure that the depth belongs to the current workpiece surface, effectively reducing the risk of incorrectly changing the texture scale due to the background measured by distance.

[0019] (3) In addition, it is possible to determine the contribution of depth to scale conversion based on depth range, sliding fluctuation and projection point inclusion relationship.

[0020] Therefore, the data processing method for material identification described above in this embodiment of the invention can achieve a low-cost solution for efficient and high-precision visual identification of material surfaces using only ordinary vision sensors (such as RGB cameras) and traditional ranging sensors (such as low-cost TOF sensors that only output distance). Thus, the target surface distance measured by single-point ranging can be used as a benchmark for converting the texture scale of monocular images, unifying the sampling window at different shooting distances to a fixed actual millimeter range. This ensures that the texture of the same material maintains an approximately consistent physical scale in the network input, reducing the classification domain shift caused by distance variations.

[0021] Specifically, in this embodiment of the invention, the calibrated ranging ray can be projected onto the camera image coordinate system, and the corresponding ranging distance is used only for the instance segmentation region containing the projected point, effectively preventing the distance from being mistakenly mapped to the background or adjacent workpieces. The ranging distance from the ranging sensor is only used for target association, depth validity determination, and texture scale conversion, and is not required as a feature for color, reflectivity, or material category. Therefore, a single-point ranging sensor (such as TOF) that only outputs distance can be used, and this ranging distance can be used to complete the physical scale normalization of the image texture and the correspondence verification between the measurement point and the target region. In the case of a single-vision image, efficient and high-precision material recognition can be achieved.

[0022] It should be understood that the above general description and the following specific embodiments are merely exemplary and illustrative, and do not limit the scope of the invention. Attached Figure Description

[0023] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0024] Figure 1 The illustrations depict application scenarios of a data processing method for material identification and a monocular vision-based material identification method, apparatus, device, medium, and program product according to embodiments of the present invention.

[0025] Figure 2A A flowchart illustrating a data processing method for material identification according to an embodiment of the present invention is shown schematically.

[0026] Figure 2B A flowchart illustrating a material identification method based on monocular vision according to an embodiment of the present invention is shown schematically.

[0027] Figure 3 The diagram illustrates an overall system flowchart of pixel scale normalization and material recognition based on monocular vision, which is a data processing method for material recognition and a monocular vision-based material recognition method according to an embodiment of the present invention.

[0028] Figure 4 This schematic diagram illustrates the principle of pixel physical scale normalization in a data processing method for material identification according to an embodiment of the present invention.

[0029] Figure 5 The illustration shows known material classification and unknown material rejection scenarios of the monocular vision-based material identification method according to an embodiment of the present invention.

[0030] Figure 6A This schematic diagram illustrates a structural block diagram of a data processing apparatus for material identification according to an embodiment of the present invention;

[0031] Figure 6B A schematic diagram illustrating the structure of a monocular vision-based material recognition device according to an embodiment of the present invention is shown; and

[0032] Figure 7 The diagram illustrates an electronic device suitable for implementing a data processing method for material identification and a monocular vision-based material identification method according to embodiments of the present invention.

[0033] The accompanying drawings mentioned above are part of the specification of embodiments of the present invention, illustrating exemplary embodiments of the invention. The drawings, together with the description in the specification, serve to illustrate the principles of the embodiments of the present invention. It should be understood that the above general description with reference to the drawings and the following detailed description are merely exemplary and illustrative, and do not limit the scope of the invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the spirit of the contents disclosed in the present invention will be clearly explained below with reference to the accompanying drawings and detailed description. After understanding the embodiments of the present invention, any person skilled in the art can make changes and modifications based on the technology taught in the present invention without departing from the spirit and scope of the present invention.

[0035] The illustrative embodiments and descriptions of the present invention are used to explain the invention, but are not intended to limit the invention. Furthermore, elements / components using the same or similar reference numerals in the drawings and embodiments are used to represent the same or similar parts.

[0036] The terms "first," "second," etc., used in this invention do not specifically refer to any order or sequence, nor are they intended to limit the invention; they are merely used to distinguish elements or operations described using the same technical terms.

[0037] The directional terms used in this invention, such as up, down, left, right, front, or back, are merely for reference to the accompanying drawings. Therefore, the directional terms used are for illustrative purposes and not for limiting the scope of this invention.

[0038] The terms “comprising,” “including,” “having,” “containing,” etc., used in this invention are all open-ended terms, meaning that they include but are not limited to.

[0039] The term "and / or" as used in this invention includes any or all combinations of the things mentioned.

[0040] In this invention, "multiple" includes "two" and "more than two"; in this invention, "multiple groups" includes "two groups" and "more than two groups".

[0041] The terms "approximately," "about," etc., used in this invention are intended to modify any quantity or error that may vary slightly, but these slight variations or errors do not change the essence of the quantity or error. Generally, the range of slight variations or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0042] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0043] When expressions such as "at least one of A, B, and C" are used, they should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). When expressions such as "at least one of A, B, or C" are used, they should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). A person skilled in the art should also understand that any conjunction and / or phrase that substantially arbitrarily indicates two or more optional items, whether in the specification, claims, or drawings, should be understood to indicate the possibility of including one of these items, either of these items, or both items. For example, the phrase “A or B” should be understood as including the possibility of “A” or “B”, or “A and B”.

[0044] In the relevant technical field, to address the problem of poor recognition accuracy, there are mainly four traditional solutions: material classification based on RGB color histograms or traditional texture descriptors; single-image material classification based on convolutional neural networks; material detection based on near-infrared, hyperspectral, Raman, or X-rays; and visual material classification that directly scales the image to a fixed network input size at unknown shooting distances.

[0045] The four traditional methods mentioned above have at least the following drawbacks: For pure RGB recognition, it is easy to confuse different materials with similar colors, and changes in shooting distance will alter the pixel scale of the texture. However, if the target area at any distance is directly scaled to the same network input size, the physical area actually observed by the network is not consistent. In addition, existing technologies typically lack the correspondence constraint between ToF measurement points and visual targets, the rejection of unknown materials, and continuous frame stabilization mechanisms, resulting in poor detection accuracy and recognition efficiency.

[0046] To address the aforementioned shortcomings of traditional material identification technologies, this invention provides a data processing method and apparatus for material identification, as described in Example 1, and a material identification method and apparatus based on monocular vision, as described in Example 2, in order to solve at least one of the aforementioned technical problems and provide a real-time material identification process with lower cost, higher efficiency, and higher accuracy.

[0047] like Figures 2A-5As shown, the methods provided in Embodiments 1 and 2 of this invention are specifically implemented by an image acquisition and data processing system based on monocular vision and single-point ranging. This system may include a monocular camera, a single-point ToF sensor that only outputs distance, a synchronous acquisition module, a camera and ToF calibration module, a target localization and association module, a physical scale normalization module, a visual classification module, an unknown material rejection module, and a time-series stabilization module. This system does not rely on ToF echo intensity or material characteristics. The synchronous acquisition module is responsible for aligning the timestamps of the RGB image and the ToF distance; the camera and ToF calibration module solves for the extrinsic parameters between the two sensors and the camera's intrinsic parameters; the target localization and association module completes candidate region detection and ToF spot-ROI association; the physical scale normalization module converts a fixed millimeter window into a pixel window based on the effective depth; the visual classification module extracts features from multi-scale texture blocks and outputs category probabilities; the unknown material rejection module determines out-of-training samples based on probability, energy, and feature distance; and the time-series stabilization module fuses the probabilities of consecutive frames of the same target and controls confirmation and reset.

[0048] As will be discussed later, in conjunction with, Figure 3 The data processing method for material identification in Embodiment 1 and the overall flow of the material identification method based on monocular vision in Embodiment 2 of the present invention include operations S301-S307.

[0049] In operation S301, coordinate system association pre-calibration is performed between the monocular vision image coordinate system and the single-point ranging sensor coordinate system, i.e., joint coordinate system calibration. In operations S321 and S322, based on the coordinate system transformation relationship of the above joint calibration, the monocular vision image (such as an RGB image) and the single-point ranging distance (such as a single-point ToF distance) are acquired simultaneously, and image preprocessing, depth conversion, and filtering are performed sequentially. In operation S303, the process of associating projection points (such as ToF projection points) with the target region is performed. In operation S304, pixel scale conversion for a fixed millimeter window is implemented. In operation S305, monocular vision classification is performed, unknown material rejection is performed in operation S306, and multi-frame probability fusion and stable output are performed in operation S307. Specifically, the above will be explained in conjunction with specific embodiments 1 and 2 below. Figure 3 The operations shown in S301-S307 will be explained in more detail.

[0050] Figure 1 The illustrations depict application scenarios of data processing methods for material identification and monocular vision-based material identification methods, apparatuses, devices, media, and program products according to embodiments of the present invention.

[0051] like Figure 1As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0052] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0053] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0054] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0055] It should be noted that the method provided in the embodiments of the present invention can generally be executed by server 105. Correspondingly, the apparatus provided in the embodiments of the present invention can generally be located in server 105. The method provided in the embodiments of the present invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the apparatus provided in the embodiments of the present invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0056] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0057] Example 1:

[0058] The following will be based on Figure 1 The described scene, through Figures 2A to 6AThe data processing method for material identification according to the disclosed embodiment 1 will be described in detail.

[0059] like Figure 2A As shown, one aspect of an embodiment of the present invention provides a data processing method for material identification, which includes operations S211 to S214.

[0060] In operation S211, for each frame of monocular visual image of the received target to be identified, a set of candidate surface regions is extracted.

[0061] In operation S212, based on the candidate set of projection points of the ranging measurement points for the target to be identified in the image coordinate system corresponding to the monocular vision image, the candidate association score corresponding to each candidate surface region in the candidate surface region set is determined.

[0062] In operation S213, at least one candidate surface region in the candidate surface region set is determined as the association target region based on the candidate association score; and

[0063] In operation S214, based on the initial ranging distance of the ranging point in the pre-calibrated ranging sensor coordinate system corresponding to the axial effective depth in the visual sensor coordinate system, and combined with the pre-calibration parameters, the pixel window data of the associated target area in the image coordinate system is determined. The pixel window data is used to realize material recognition.

[0064] like Figures 2A-6B As shown, according to an embodiment of the present invention, before operation S211 extracts a set of candidate surface regions for each frame of the received monocular visual image of the target to be identified, the method further includes:

[0065] Based on the joint calibration relationship between the visual sensor and the ranging sensor, the visual sensor acquires each frame of the original monocular image of the target to be identified, and the ranging sensor simultaneously acquires the initial ranging distance of the target to be identified based on the ranging measurement point.

[0066] Each frame of the original monocular image is preprocessed to generate a monocular vision image; and the initial ranging distance is processed based on the joint calibration relationship to generate the original axial depth of the corresponding ranging measurement point in the visual sensor coordinate system.

[0067] In this embodiment of the invention, the visual sensor can be an image acquisition sensor, such as an RGB camera or an image detection sensor (such as a CCD), i.e., an image sensor. The ranging sensor can be a distance sensor that locates and measures the distance of an object based on light, electromagnetic waves, sound waves, or ultrasound, such as ranging rays, laser ranging, etc., specifically such as a time-of-flight ranging sensor.

[0068] like Figure 3 As shown, in operation S301, the joint calibration relationship can be based on the joint calibration of the coordinate systems of the vision sensor and the ranging sensor. Specifically, it can be implemented based on the pre-calibration parameters of each sensor. The pre-calibration parameters may include intrinsic parameters of the vision sensor (such as camera intrinsic parameters) and extrinsic parameters of the ranging sensor from the ranging sensor coordinate system to the vision sensor coordinate system, etc., which will not be elaborated here. Therefore, based on this joint calibration of the coordinate systems, it can be ensured that the subsequent coordinate transformation between the ranging distance and the monocular vision image is more accurate.

[0069] In operation S321, the visual sensor can perform visual detection on the target to be identified to obtain at least one frame of raw monocular image. The target to be identified can be an object to be identified, such as various objects or living beings (including other embodied intelligences) in the environment of the embodied intelligence, such as furniture in a home environment or workpieces in an industrial manufacturing environment. During the acquisition of the monocular visual image, the ranging sensor can simultaneously measure the distance to the target to be identified and obtain the single-point ranging distance as the initial ranging distance, as in operation S322.

[0070] For each frame of the original monocular image, an image preprocessing process is performed, which may include image distortion correction, white balance, and color chart calibration, among other things, without explicit limitations. After this image preprocessing, a monocular visual image corresponding to the specified frame number is generated; this monocular visual image is a distortion-corrected image. Correspondingly, depth processing can be performed on the initial ranging distance simultaneously to generate the original axial depth of the ranging measurement point in the visual sensor coordinate system, conforming to the joint calibration relationship. Therefore, the original axial depth can be understood as the pixel-level depth mapped from the initial ranging distance to the visual sensor coordinate system, such as the axial depth Z in the camera coordinate system.

[0071] The image coordinate system can correspond to the image pixel coordinate system of the monocular vision image of the vision sensor, and can be used to map three-dimensional space to image pixels for two-dimensional representation. The vision sensor coordinate system can be the spatial coordinate system of the image acquisition sensor itself, such as the coordinate system of a three-dimensional camera based on an RGB camera. The ranging sensor coordinate system can be the coordinate system corresponding to the initial ranging distance of the ranging sensor, such as the coordinate system of a ToF sensor. The joint calibration relationship can be understood as the coordinate mapping relationship between the vision sensor coordinate system and the ranging sensor coordinate system. For example, the ranging distance under the ranging sensor coordinate system can be uniformly mapped to the vision sensor coordinate system. The ranging measurement point can be understood as the hit point on the surface of the target to be identified (such as a workpiece) on which the ranging sensor emits a ranging ray (such as light, electromagnetic wave, etc.) to obtain the initial ranging distance. The initial ranging distance is the distance from the ranging sensor to the hit point along the ranging ray. Typically, in the embodiments of this invention, when the vision sensor performs detection of the original monocular image, it may include the synchronous acquisition of the ranging measurement point.

[0072] like Figures 2A-6B As shown, according to an embodiment of the present invention, after performing image preprocessing on each frame of the original monocular image to generate a monocular visual image; and performing depth processing on the initial ranging distance based on the joint calibration relationship to generate the original axial depth of the corresponding ranging measurement point in the visual sensor coordinate system, the method further includes:

[0073] Based on the pixel projection points of the monocular visual image corresponding to the joint calibration relationship in the visual sensor coordinate system, determine the projection coordinate points of the ranging measurement point in the visual sensor coordinate system.

[0074] The projection pixel coordinates of the ranging measurement point in the corresponding monocular vision image are determined based on the projection coordinates and the intrinsic parameters of the vision sensor with pre-calibrated parameters.

[0075] like Figure 3 As shown, for the process of associating the projection point and target area in operation S303, after completing the joint calibration of the coordinate system and obtaining the corresponding monocular visual image and the original axial depth, the axial depth mapping relationship between the visual sensor coordinate system and the ranging sensor coordinate system can be further determined.

[0076] First, the camera (visual sensor) intrinsic parameter matrix is: The main point is Focal length is and distortion parameters are The focal length is measured in pixels, and the distortion parameter D can be used for image distortion correction. Any pixel in a monocular vision image... At this original axial depth (e.g., camera axial depth) The back projection point at () can be expressed as:

[0077] (1)

[0078] Among them, the back projection point It can be understood as a three-dimensional point in the camera coordinate system, that is, a pixel projection point; , These are the two components of the pixel p coordinate; , The pixel coordinates of the camera's principal point; , The camera focal length (in pixels); This is the camera axial depth corresponding to that pixel, i.e., the original axial depth.

[0079] Based on the back projection point of the above formula (1), let the unit measurement ray in the ToF coordinate system be... Then the extrinsic parameters from the ToF coordinate system (i.e., the range sensor coordinate system) to the camera coordinate system (i.e., the vision sensor coordinate system) are: Therefore, when the initial ranging distance is measured along this measuring ray by a ranging sensor (such as a ToF sensor), Then, the mapped coordinates of the corresponding ranging point in the camera coordinate system (i.e., the vision sensor coordinate system) can be expressed as:

[0080] (2)

[0081] in, It can be understood as the coordinates of the ranging measurement point in the visual sensor coordinate system (such as the camera coordinate system), that is, the three-dimensional projected coordinate point, such as the ToF hit point P_c^{tof}=(X_t,Y_t,Z_t)^T in the camera coordinate system; This is the rotation matrix from the range sensor coordinate system to the vision sensor coordinate system; For the corresponding translation vector; The unit measurement ray in the distance sensor coordinate system ; The ranging sensor measures the actual physical distance between the target to be identified and the ranging measurement point along the measuring ray.

[0082] Therefore, let the coordinates of the ranging point in the visual sensor coordinate system be... and Then, the projected pixels of the ranging point in the distortion-free monocular vision image can be expressed as:

[0083] (3)

[0084] in, This can be understood as the two-dimensional projection pixel coordinates of the ranging measurement point in the distortion-free monocular vision image; π(·) is the pinhole camera projection function; This is the camera intrinsic parameter matrix (i.e., the visual sensor intrinsic parameters). , , These are the projection coordinates of the points. The three components in the camera coordinate system .

[0085] It should be noted that, in the embodiments of the present invention, [the following is used] As the camera axial depth required for scale conversion, the ToF slant distance cannot be used. or relative to the height of the workbench Directly substitute the pinhole model. If the system first calculates the workpiece height from the empty worktable reference distance... Then you should press again. Conversion, among which Let h be the axial depth from the camera's optical center to the worktable's reference surface, and h be the height of the workpiece relative to the worktable surface. In this embodiment of the invention, the projected pixel coordinates of the ranging point in the monocular vision image are required. Located within the current candidate target region. Only when the camera and ToF baseline are negligible and the projection drift over the entire working distance has been verified to be within tolerance can [the target region be selected]. Approximate to a fixed pixel value; otherwise, use the current value. The extrinsic parameters are calculated frame by frame. The above conversion also assumes that the ToF hit point and the clipped texture block are located on the same approximate plane.

[0086] Therefore, based on the joint calibration relationship between the aforementioned coordinate systems, it is possible to achieve pixel mapping of the ranging physical parameters such as the ranging measurement point and the initial ranging distance relative to the monocular vision image. This can effectively ensure the projection mapping between the ranging spot and the target area in the subsequent data processing process, and realize the association between the projection point and the target area.

[0087] like Figures 2A-6B As shown, according to an embodiment of the present invention, after performing image preprocessing on each frame of the original monocular image to generate a monocular visual image; and performing depth processing on the initial ranging distance based on the joint calibration relationship to generate the original axial depth of the corresponding ranging measurement point in the visual sensor coordinate system, the method further includes:

[0088] Median filtering is performed on the original axial depth to determine the filtered axial depth of the corresponding distance measurement point.

[0089] Exponential smoothing is performed on the filtered axial depth to determine the smoothed axial depth of the corresponding ranging measurement point.

[0090] The validity of the smoothed axial depth of the distance measurement point corresponding to the monocular vision image is determined, and the mean of at least one historical valid depth and the smoothed axial depth determined by the validity is processed to determine the axial valid depth in the vision sensor coordinate system.

[0091] like Figure 3 As shown, in operations S322-S303, further depth filtering and validity processing of the initial ranging distance of the ranging sensor can be implemented.

[0092] The median filtering process for the original axial depth of each frame mapped to the camera coordinate system can be expressed as follows:

[0093] (4)

[0094] in, The axial depth of the k-th frame after median filtering is the filtered axial depth. This represents the original axial depth of the k-th frame; This is the length of the median filter window.

[0095] Furthermore, the smoothing process for the filtering axial depth of each frame in the above formula (4) can be expressed as follows:

[0096] (5)

[0097] in, The axial depth of the k-th frame after exponential smoothing is the smoothed axial depth. The preset smoothing coefficient is 0 < β ≤ 1; This is the smoothed result of the previous frame, i.e., the smoothed axial depth of the (k-1)th frame.

[0098] Furthermore, regarding the depth validity and multi-physical-scale texture sampling process, considering that the initial ranging distance obtained by the ranging sensor can be used to generate the original axial depth in the camera coordinate system, without establishing echo or material feature branches, the smoothed axial depths of the most recent M that have passed the initial validity check and completed the above filtering and smoothing process can be used. The mean value is calculated as follows:

[0099] (6)

[0100] in, σ_{Z,k} is the mean of the effective smoothing depths (referred to as historical effective depths) of the most recent M historical frames, i.e., the axial effective depth; σ_{Z,k} is the corresponding standard deviation. is the smooth axial depth of the ki-th frame after validity determination.

[0101] This allows for the realization of the effective axial depth of the visual sensor coordinate system based on depth conversion and filtering smoothing, thereby ensuring that the scale conversion has more accurate effective axial depth parameters.

[0102] Therefore, by first converting the original ToF distance (i.e., the initial ranging distance) into the camera axial depth... (i.e., the original axial depth), then perform median filtering and exponential smoothing on the samples that pass the initial detection based on range and geometric correlation, and do not update invalid samples. This allows for direct scaling of the data from the nearest source in subsequent scale conversions. Effective smoothing axial depth The obtained axial effective depth In the subsequent region segmentation selection process, if the depth exceeds the limit, the fluctuation exceeds the limit, or the continuous jump exceeds the limit, or... If the target is not within the candidate target, no scale conversion will be performed in this frame; the system can only retain the previous valid result within a limited number of frames, or output "Depth invalid, not recognized at this time".

[0103] like Figures 2A-6B As shown, according to an embodiment of the present invention, in operation S211, for each frame of the received monocular visual image of the target to be identified, the extraction of a candidate surface region set includes:

[0104] Detection and segmentation are performed on monocular vision images to extract multiple candidate surface regions as a set of candidate surface regions.

[0105] For segmentation of monocular vision images after image preprocessing such as distortion correction, candidate region segmentation can be performed on each frame of the monocular vision image through methods such as object detection or instance segmentation to generate multiple candidate surface regions. This is used to construct a set of candidate surface regions. The specific process for extracting the target region in image detection and segmentation is not limited. In some embodiments, candidate region segmentation can be understood as the segmentation process of the Region of Interest (ROI).

[0106] like Figures 2A-6B As shown, according to an embodiment of the present invention, in operation S212, based on the candidate set of projection points of the ranging measurement points for the target to be identified in the image coordinate system corresponding to the monocular vision image, determining the candidate association score corresponding to each candidate surface region in the candidate surface region set includes:

[0107] Based on the candidate set of projection points corresponding to the candidate surface regions containing the projected pixel coordinates, and combined with preset normalization parameters, the candidate association score corresponding to each candidate surface region is determined.

[0108] like Figure 3 As shown, in operation S303, the projection point-target area association is performed to achieve a precise association between target location and target area.

[0109] Candidate surface regions can be obtained through target detection or instance segmentation as described above. The candidate surface region set is constructed, and a ToF projection point (projected pixel coordinates) is constructed. Candidate set of projection points For each candidate surface region in the candidate surface region set... (i.e., j) calculates candidate association scores. The judgment is as follows (where j∈J_t):

[0110] (7)

[0111] in, The candidate correlation score for candidate surface region j; The instance segmentation region for the j-th candidate surface target is the candidate surface region; The projected pixel coordinates of the distance measurement point The candidate set, i.e., the candidate set of projection points; The detection confidence level is normalized to [0,1]. To ensure the rationality of normalized area; Penalty for boundary anomalies; As a candidate center; The length of the candidate box diagonal; To prevent small positive numbers from being divided by zero; All weights are non-negative. The preset normalization parameters may include detection confidence, normalized area reasonableness, boundary anomaly penalty parameters, candidate centers, small positive numbers, and non-negative weights.

[0112] Therefore, it is possible to associate the image projection points of the measurement rays based on the ranging points with the visual candidate regions, ensuring that the measured depth belongs to the surface of the target to be identified. For example, the ToF projection point-target region association can ensure that the ranging depth belongs to the surface to be identified.

[0113] like Figures 2A-6B As shown, according to an embodiment of the present invention, in operation S213, determining at least one candidate surface region in the candidate surface region set as the associated target region based on the candidate association score includes:

[0114] The candidate surface regions are sorted according to their candidate association scores to generate a candidate region set.

[0115] The candidate surface region with the largest candidate association score in the candidate region set is selected as the association target region.

[0116] As mentioned earlier, a candidate association score can be determined for each candidate surface region. This score confirms the degree of association between the corresponding ranging projection point and the candidate surface region. Generally, the higher the candidate association score, the higher the association between the ranging projection point and the candidate surface region. Therefore, by using the magnitude of this candidate association score, each candidate surface region in the candidate surface region set can be sorted to generate a candidate region set with a ranking rule based on the candidate association score.

[0117] Select the candidate surface region with the highest candidate correlation score from the candidate region set as the correlation target region. When the candidate region set J_t is empty, the Time-of-Flight (ToF) distance can be omitted for scale conversion. This ensures the most realistic and accurate relationship between the ranging sensor coordinate system and the vision sensor coordinate system.

[0118] like Figures 2A-6B As shown, according to an embodiment of the present invention, in operation S214, based on the initial ranging distance of the ranging point in the pre-calibrated ranging sensor coordinate system corresponding to the axial effective depth in the visual sensor coordinate system, and in combination with pre-calibration parameters, the pixel window data of the associated target region in the image coordinate system is determined, including:

[0119] Based on the physical size information of the preset actual texture window corresponding to the associated target area, the pixel focal length parameters of the pre-calibrated parameters, and the effective axial depth, the pixel size information of the preset actual texture window in the image coordinate system is determined as pixel window data.

[0120] like Figure 3 and Figure 4 As shown, a fixed millimeter window pixel scale conversion is performed in operation S304 to achieve physical scale normalization based on axial depth.

[0121] In this embodiment of the invention, the texture period of the same material occupies different numbers of pixels at different distances, such as... Figure 4 As shown, the same physical texture occupies more pixels at close range and fewer pixels at greater distances. Therefore, for a candidate surface region that has been distortion-corrected, has a locally planar surface, and is approximately directly facing the camera (i.e., at least one associated target region identified above), let the actual texture window be... And the effective depth along the camera axis is The corresponding pixel window data contains the following pixel size information:

[0122] (8)

[0123] (9)

[0124] in, This represents the width and height of the pixel window corresponding to the associated target region; This sets the width and height (in millimeters or meters) of the actual texture window, i.e., the physical dimensions of the actual texture window; the focal length is... (Unit: pixels); Z is the effective depth along the camera axis, taken as... , which is the effective axial depth of the associated target region; round(·) is rounding to the nearest integer; Ω is the target mask region, i.e., the associated target region; For fixed network input dimensions. It uses the same unit of length as Z. Pre-calibration parameters may include focal length, fixed net input size, etc.

[0125] This invention selects only those elements that can be completely contained within the target mask Ω. Rectangular texture blocks are then resampled to a fixed network input size. .like If the number of pixels is less than the preset minimum, exceeds the target's usable range, or the texture block crosses the target boundary, the scale is discarded without clipping and intersection to avoid altering the predetermined actual millimeter range. When surface tilt is not negligible, perspective correction is required based on the known planar homography matrix, surface normal, or additional depth information; single-point ToF cannot independently determine the pose of an unknown surface. If the aforementioned prior information cannot be obtained, only texture blocks approximately facing the camera area are used, or the texture window is reduced to decrease the scale error caused by tilt.

[0126] Therefore, the camera axial depth obtained by single-point ranging can normalize the texture blocks of monocular vision images to a fixed actual millimeter scale. Specifically, it can be determined whether the depth can participate in the scale conversion based on the depth range, sliding fluctuation and projection point inclusion relationship. Finally, pixel scale conversion based on a fixed millimeter window is realized, so that the textures at different distances have a consistent physical scale. This can effectively ensure the accuracy and efficiency of surface material recognition based on monocular vision. See Example 2 below for details.

[0127] Based on the above-described data processing method for material identification, this invention also provides a data processing apparatus for material identification. The following will be combined with... Figure 6A The device is described in detail.

[0128] Figure 6A A schematic block diagram of a data processing apparatus 610 for material identification according to an embodiment of the present invention is shown.

[0129] like Figure 6A As shown, the data processing device 610 for material identification in this embodiment includes a region extraction module 611, a score determination module 612, a region determination module 613, and a window determination module 614.

[0130] The region extraction module 611 is used to extract a set of candidate surface regions for each frame of the received monocular visual image of the target to be identified. In one embodiment, it can be used to perform the operation S211 described above, which will not be repeated here.

[0131] The scoring determination module 612 is used to determine the candidate association score corresponding to each candidate surface region in the candidate surface region set based on the candidate set of projection points of the ranging measurement points for the target to be identified in the image coordinate system corresponding to the monocular vision image. In one embodiment, it can be used to perform the operation S212 described above, which will not be repeated here.

[0132] The region determination module 613 is used to determine at least one candidate surface region from the candidate surface region set as the association target region based on the candidate association score. In one embodiment, it can be used to perform the operation S213 described above, which will not be repeated here.

[0133] The window determination module 614 is used to determine the pixel window data of the associated target area in the image coordinate system based on the axial effective depth in the visual sensor coordinate system corresponding to the initial ranging distance of the ranging point in the pre-calibrated ranging sensor coordinate system, and in combination with the pre-calibration parameters. The pixel window data is used to realize material recognition. In one embodiment, it can be used to perform the operation S214 described above, which will not be repeated here.

[0134] According to embodiments of the present invention, any plurality of modules among the region extraction module 611, score determination module 612, region determination module 613, and window determination module 614 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the region extraction module 611, score determination module 612, region determination module 613, and window determination module 614 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any one of the three implementation methods or a suitable combination of any of them. Alternatively, at least one of the region extraction module 611, the score determination module 612, the region determination module 613, and the window determination module 614 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0135] Example 2:

[0136] The following will be based on Figure 1 The described scenario, combined with the data processing method and apparatus for material identification in Embodiment 1 above, is used through... Figures 2B to 6B The material recognition method based on monocular vision in Embodiment 2 is described in detail.

[0137] like Figure 2B As shown, one aspect of an embodiment of the present invention provides a material identification method based on monocular vision, which includes operations S221 to S224.

[0138] In operation S221, visual embedding features of the associated target region are extracted based on the monocular visual image of the target to be identified and the pixel window data of the associated target region corresponding to the ranging measurement point.

[0139] In operation S222, visual aggregate features of visual embedding features are generated by associating the visual quality scores of the target region with aggregated weight data; and

[0140] In operation S223, the probability of the material category of the target to be identified for a preset category attribute is obtained by visual aggregation features;

[0141] In operation S224, the target category attribute of the target to be identified is obtained based on the material category probability and the pre-generated material rejection output judgment condition, and the material identification is completed.

[0142] Based on the data processing method and apparatus for material identification in Embodiment 1, pixel scale conversion based on a fixed millimeter window can be achieved by using the pixel size information of the associated target region window (i.e., pixel window data), so that textures at different distances have a consistent physical scale. Furthermore, based on this, it is possible to further realize... Figure 3 Operations S305-S307 are shown to achieve material identification of the target object. In operation S305, the process of monocular vision classification is first considered, specifically including the extraction of visual embedding features of the associated target region corresponding to the monocular vision image. Therefore, based on the aforementioned pixel scale conversion, these visual embedding features can more realistically reflect the true visual information of the target object, such as texture, color, edges, and gloss, thus greatly improving the accuracy of subsequent material identification. The visual embedding features can be feature vector representations of this embedding information, which will not be elaborated further.

[0143] like Figures 2B-6B As shown, according to an embodiment of the present invention, in operation S221, based on the monocular visual image of the target to be identified and the pixel window data of the associated target region corresponding to the ranging measurement point, the visual embedding features of the associated target region are extracted, including:

[0144] For the normalized texture block extracted from the pixel window data of the associated target region, the corresponding color visual features, texture visual features, edge visual features and gloss visual features are extracted for each pixel using a preset visual extractor, and the region-level color moment features and gray-level co-occurrence moment features are extracted for the normalized texture block.

[0145] Normalized texture blocks are texture regions corresponding to visual embedding features, specifically determined by pixel window data of the associated target region calculated by pixel scale. The preset visual extractor can be an extraction architecture based on visual feature vectors from backbone networks such as convolutional neural networks or visual transformers, and can be used to extract visual embedding information such as color, texture, edge, and gloss from monocular vision images.

[0146] like Figures 2B-6B As shown, according to an embodiment of the present invention, the gloss visual feature includes joint specular features. In the normalized texture block extracted from the pixel window data of the associated target region, a preset visual extractor extracts the corresponding color visual features, texture visual features, edge visual features, and gloss visual features for each pixel. Furthermore, the normalized texture block is used to extract region-level color moment features and gray-level co-occurrence moment features, including:

[0147] Based on the pixel brightness value Y(x,y), pixel saturation S(x,y), and local gradient magnitude of the pixel. Binary mask for specular candidate regions corresponding to normalized texture blocks Determine joint specular features Among them, joint specular features Including the proportion of highlight area in the highlight candidate region Pixel count Average brightness Average saturation and average local gradient .

[0148] like Figure 3 As shown, in operation S305, for monocular vision classification, a visual feature extraction process is performed. This visual feature extraction primarily targets the normalized texture blocks corresponding to the pixel window data of the associated target region mentioned in Example 1. These normalized texture blocks have already undergone pixel scale transformation, unifying the sampling windows at different shooting distances into a fixed actual millimeter-range visual image region. This ensures that the texture of the same material maintains an approximately consistent physical scale in the network input, reducing classification domain shift caused by distance variations.

[0149] Therefore, after the original visual image is used as the input for the visual branch, a monocular visual image (such as an RGB / Lab image) is generated through white balance or color card correction. This image is then normalized to the physical scale as described in Embodiment 1 to form a normalized texture block defined by n_x and n_y of the associated target region. This normalized texture block corresponds to the target mask region Ω in Embodiments 1 and 2 of this invention. For the monocular visual image corresponding to this normalized texture block, a preset visual extractor composed of a convolutional neural network or a visual Transformer can be used to extract the color visual features, texture visual features, edge visual features, and gloss visual features corresponding to each pixel on the texture block as visual embedding features f_v. In some embodiments, an auxiliary visual feature extraction process, including color moment features and gray-level co-occurrence matrix features, can be performed on the normalized texture block to unify the components of the visual embedding feature f_v. Those skilled in the art should understand that color visual features, texture visual features, and edge visual features are not specifically limited here. In addition, color moment features and gray-level co-occurrence matrix features are regional statistics of texture blocks, specifically regional features extracted from normalized texture blocks.

[0150] In this embodiment of the invention, the color moment feature can be characterized as follows:

[0151] (10)

[0152] (11)

[0153] Where μ_{I,c} and σ_{I,c}² are the mean and variance of the color moments of image I in channel c, as defined by the normalized texture block; R is a preset set of pixel indices, and |R| is the number of its corresponding elements; I_{i,c} is the value of the i-th pixel in channel c.

[0154] Similarly, the gray-level co-occurrence moment feature can be characterized as follows:

[0155] (12)

[0156] (13)

[0157] Where Contrast and Homogeneity are the contrast and homogeneity of the gray-level co-occurrence matrix features, respectively; P(i,j) is the normalized value of the gray-level co-occurrence matrix at (i,j).

[0158] In this embodiment of the invention, the gloss visual feature may further include a joint specular feature. For any pixel (x,y) within the target mask Ω corresponding to the aforementioned normalized texture module, a specular candidate region is first determined based on the brightness threshold and saturation threshold, and then the local gradient of the region is statistically analyzed to form a joint specular feature. Specifically, it can sequentially represent the pixel brightness value Y(x,y), pixel saturation S(x,y), and local gradient magnitude G. l The binary mask M for the specular candidate region corresponding to (x,y) and the normalized texture block. h (x,y) is as follows:

[0159] (14)

[0160] (15)

[0161] (16)

[0162] (17)

[0163] Where Y(x,y) is the pixel brightness; R(x,y), G(x,y), and B(x,y) are the red, green, and blue channel values ​​with the same range of values, respectively. and These represent the maximum and minimum values ​​of the three color channels, respectively; ε is a very small positive number to prevent the denominator from being zero; S(x,y) is the saturation. , For local gradient operators in the horizontal and vertical directions, the Sobel operator is preferred; Represents convolution; τ is the local gradient magnitude; τY and τS are the brightness threshold and saturation threshold, respectively; 1[·] is the indicator function; This is a binary mask for the candidate specular highlight region. In this embodiment of the invention, luminance Y(x,y) and saturation S(x,y) are used to determine the main specular highlight and the local gradient. It is used to characterize the boundaries of highlights and changes in brightness, and to avoid incorrectly excluding highlights due to their small internal gradients.

[0164] Among them, according to the aforementioned formulas (14)-(17), the proportion of the highlight area of ​​the highlight candidate region is further represented. Pixel count Average brightness Average saturation and average local gradient as follows:

[0165] (18)

[0166] (19)

[0167] (20)

[0168] (twenty one)

[0169] (twenty two)

[0170] Where |Ω| represents the total number of pixels within the target mask; The percentage of the area covered by highlights; The number of pixels in the highlight candidate region; , and These represent the average brightness, average saturation, and average local gradient of the highlight candidate region, respectively.

[0171] Therefore, combining the above formulas (18)-(22), the joint specular feature can be further expressed as follows:

[0172] (twenty three)

[0173] in, This is the joint specular feature vector.

[0174] In some embodiments of the present invention, the number of specular regions, the area of ​​connected regions, and shape features can be further statistically analyzed, and the joint specular feature vector f is calculated. hIt is concatenated with the visual embedding feature f_v to help distinguish between metals, coated plastics, and matte materials. The above threshold can be determined by a calibration set or adaptively determined based on the brightness and saturation statistics of the target region.

[0175] like Figures 2B-6B As shown, according to an embodiment of the present invention, before extracting the corresponding color visual features, texture visual features, edge visual features, gloss visual features, color moment features, and gray-level co-occurrence moment features for each pixel point from the normalized texture block extracted from the pixel window data of the associated target region using a preset visual extractor, the method further includes:

[0176] Extract the actual pixel region of the associated target region based on the pixel window data;

[0177] The actual pixel area is resampled at a preset network input size to generate multi-scale image blocks as normalized texture blocks.

[0178] As mentioned above, a normalized texture block can be understood as the surface area of ​​the pixel window data corresponding to the associated target region, which is specifically related to its depth validity and multi-physical scale texture sampling.

[0179] As mentioned in Example 1 above, a ranging sensor (such as a ToF sensor) can be used to generate camera axial depth without typically establishing echo or material feature branches. Therefore, the most recent M depths that have passed initial detection and completed filtering are used... In other words, the corresponding mean processing result can be expressed as:

[0180] (twenty four)

[0181] in, σ_{Z,k} is the mean of the most recent M effective smoothing depths, i.e., the axial effective depth; σ_{Z,k} is the corresponding standard deviation. Let be the smoothing depth of the ki-th frame.

[0182] Accordingly, when the effective depth of this axis satisfy: When the most recent M samples are all valid and the projected pixel coordinates q_t of the aforementioned ranging point in the monocular vision image are located within the associated target region, the depth can be used for scale conversion. In this case, referring to formulas (8) and (9), the pixel window data determined in Example 1 above can be used. For J preset actual windows Calculate separately , This results in multi-scale image patches ranging from fine textures to overall texture, as follows:

[0183] (25)

[0184] Where R_j is the actual cropped pixel region in the image, i.e., the actual pixel region, which must fall completely within the target mask Ω; n_{x,j} and n_{y,j} are the pixel window sizes corresponding to the j-th scale, corresponding to the pixel window data of the aforementioned associated target region.

[0185] Furthermore, by resampling the actual pixel region R_j to a size that conforms to the preset network input size, the corresponding multi-scale image patch can be determined as a normalized texture patch as follows:

[0186] (26)

[0187] Where I[R_j] is the image cropping of the actual pixel region R_j; Resize(·) is the resampling function; N_in is the fixed network input side length, i.e. the preset network input size; J_v is the effective scale set that satisfies the minimum number of pixels, completely falls into the target mask, and has qualified image quality; W_j and H_j are the width and height of the j-th preset actual window.

[0188] Therefore, since depth only determines the pixel size of the texture block and is not directly input into the material classifier, it can effectively avoid the high misidentification rate caused by the accidental correlation between workpiece height or acquisition distance and material category. Specifically, when J_v is empty, the system either does not output anything or enters a separately trained and labeled model without distance backoff.

[0189] like Figures 2B-6B As shown, according to an embodiment of the present invention, before operation S222 generates visual aggregate features of visual embedding features by using aggregate weight data corresponding to the visual quality scores of the associated target regions, the method further includes:

[0190] The visual quality score is determined based on the sharpness, exposure value, and effective area of ​​the normalized texture blocks in the associated target region.

[0191] Aggregate weight data is determined based on visual quality scores and the effective scale set of normalized texture blocks.

[0192] like Figure 3 As shown, the monocular vision classification operation S305 further involves the execution process of monocular vision multi-scale feature aggregation and material classification.

[0193] The visual quality score s_j mainly targets the visual quality of the image surface region defined by the normalized texture block. This visual quality can be determined by the sharpness, exposure value, and effective area of ​​the image surface region. The explanation of image sharpness, exposure value, and effective area will not be elaborated here.

[0194] Furthermore, a shared or independent visual encoder F is used to extract the normalized texture blocks described above from each effective physical scale texture block. The feature f_j=F(X_j) is combined with the aforementioned visual quality score s_j to perform a visual feature aggregation process.

[0195] Among them, the effective scale set based on normalized texture blocks The aggregated weight data, determined together with the aforementioned visual quality score s_j, are as follows:

[0196] (27)

[0197] Here, a_j directly reflects the aggregation weight of the j-th scale; s_j is its visual quality score, which can be determined by sharpness, exposure, and effective area inside the target.

[0198] Therefore, for operation S222, visual aggregate features of visual embedding features are generated by associating the visual quality scores s_j of the target region with the aggregated weight data a_j. In other words, the corresponding visual aggregation features This can be expressed as follows:

[0199] (28)

[0200] Where f is the aggregated visual embedding, i.e., the visual aggregated feature; f_j is the visual embedding feature extracted by the visual encoder F at the j-th scale; and J_v is the aforementioned effective scale set.

[0201] The visual quality score s_j can be calculated solely from visual quality and visual features, without including range echoes (such as ToF echoes), distance, or quality features. Therefore, attention, feature stitching, or probability averaging can be used to achieve multi-physical-scale aggregation, ensuring that monocular visual classification can achieve more accurate material discrimination on normalized texture blocks.

[0202] like Figures 2B-6B As shown, according to an embodiment of the present invention, in operation S223, obtaining the probability of the material category of the target to be identified for a preset category attribute by visually aggregating features includes:

[0203] Based on the visual aggregation feature f and the logical output value z_c of the preset classifier for the preset category attribute, the material category probability is generated. .

[0204] like Figure 3 As shown, in the monocular vision classification operation S305, based on the aforementioned visual feature aggregation process, the material category probability of the final normalized texture block can be obtained as follows:

[0205] (29)

[0206] Wherein, the material category probability p(y=c|f) can be the posterior probability of material category c (i.e., the preset category attribute) given a visual embedding f (visual aggregation feature); r is the preset summation index of material categories; z_c is the classification logic value output by the preset classifier for the r-th material category (corresponding to the preset category attribute c); z_c is the logic output value (logit) of the preset classifier for this category c (preset category attribute); T is the predetermined temperature calibration parameter, where T>0, which can be determined on the independent validation set; C is the total number of known material categories, which may include category c.

[0207] In particular, when the axial depth (i.e., ToF depth) corresponding to the ranging sensor is invalid, the image that has not been physically scaled can be discarded and mixed with the normal input, thereby enabling the output to be temporarily suspended or the fallback model that has been trained separately and marked in the output to be invoked.

[0208] like Figures 2B-6B As shown, according to an embodiment of the present invention, in operation S223, obtaining the probability of the material category of the target to be identified for a preset category attribute by visually aggregating features includes:

[0209] Based on the cross-entropy loss and focus loss corresponding to the material category probabilities of the preset training samples, the preset target model architecture is pre-trained, and the model is trained and verified by the pre-generated material rejection output judgment conditions corresponding to the material category probabilities of the preset training samples, thereby generating a target material recognition model. The target material recognition model outputs the material category probability of the target to be recognized for the preset category attribute according to the input visual aggregation features, thereby realizing material recognition.

[0210] like Figure 3 As shown, in the monocular vision classification performed by operation S305, the material category probability based on the aforementioned visual aggregation feature f is combined. This allows for further training of the target model and handling of class imbalance.

[0211] For materials of class C, cross-entropy loss with class weights can be used, as shown below:

[0212] (30)

[0213] Where L_cls is the cross-entropy loss with category weights; w_c is the weight of category c (preset category attribute); y_c is the one-hot encoded label of category c (preset category attribute); and p_c is the predicted probability for category c in the aforementioned formula (29), i.e., the material category probability. = p_c. Wherein, the preset category attribute can be a pre-generated material category name (such as metal, wood, etc.) or the code or code of the corresponding material category (such as 001 for metal, 002 for wood, etc.).

[0214] Correspondingly, the focus loss can be expressed as follows:

[0215] (31)

[0216] Where L_focal is the focus loss; γ is the focus parameter, used to adjust the degree of weight decay of easily separated samples; the other symbols w_c, y_c, and p_c have the same meaning as above.

[0217] The material category probabilities involved in the aforementioned cross-entropy loss and focus loss can be known material category recognition probabilities for preset training samples (i.e., training sample recognition probabilities). Based on these training samples, a model is trained on a preset target model architecture. Combined with pre-generated material rejection output judgment conditions, the target material recognition model, after data training, can possess the efficient and accurate material recognition capabilities described in operations S221 to S224 of this embodiment. It should be noted that, without using the target material recognition model, the corresponding material recognition effect can also be achieved based on the methods of embodiments 1 and 2 of this invention.

[0218] like Figures 2B-6B As shown, according to an embodiment of the present invention, a material rejection output judgment condition is generated based on the maximum predicted probability for a preset category attribute, the input energy logic value, and / or the aggregated feature distance.

[0219] like Figure 3 As shown, in operation S306, unknown material rejection can be further introduced to realize material identification constraints, thereby further improving the accuracy of material identification.

[0220] Among them, the rejection of unknown materials can avoid forcibly classifying materials outside the training set into known categories. Specifically, the maximum probability of the model can be calibrated on the validation set as the maximum prediction probability, the energy value as the input energy logic value, and / or the feature distance threshold as the aggregated feature distance. Based on the aforementioned material category probability p_c, the following rejection conditions for unknown materials can be used individually or in combination as material rejection output judgment conditions.

[0221] Specifically, for the identification of category c, the maximum prediction probability of the preset target model architecture can satisfy:

[0222] max_c p_c<τ_p (32)

[0223] Where max_c p_c can be the maximum material class probability; τ_p can be the probability parameter threshold determined by the pre-generated independent validation set.

[0224] Similarly, for the identification of category c, the corresponding input energy logic value satisfies:

[0225] E(x)=-T log Σ_c exp(z_c / T)>τ_E (33)

[0226] Where E(x) is the energy value corresponding to input x; T is the temperature calibration parameter; z_c is the logit value of category c; τ_E is the preset energy rejection threshold, or it can be an energy parameter threshold pre-determined by a pre-generated independent validation set. Input x is the visual sample to be identified in the input material recognition model, which is a pre-processed image of the workpiece surface or a target image region.

[0227] Furthermore, for the identification of category c, the corresponding aggregated feature distance (such as Mahalanobis distance) can satisfy:

[0228] min_c (f-m_c)ᵀ(Σ_c+λI)^{-1}(f-m_c)>τ_d, λ>0 (34)

[0229] Where f is the visual aggregate feature of the aforementioned sample to be discriminated; m_c and Σ_c are the feature mean and covariance of the known category c in the training set, respectively; λI is the regularization term, λ>0, and I is the identity matrix; τ_d is the feature distance rejection threshold, which can also be a distance parameter threshold predetermined from a pre-generated independent validation set.

[0230] It should be noted that τ_p, τ_E, and τ_d are all determined by the independent validation set.

[0231] like Figure 5 As shown, in this embodiment of the invention, at least one of the above-mentioned maximum prediction probability, input energy logic value, and / or aggregated feature distance can be used as the material rejection output judgment condition to achieve the corresponding material rejection effect. For example, when any of the conditions of the maximum category prediction probability value < τ_p and the feature distance value > τ_d as the aggregated feature distance are met, rejection is enabled and unknown material / or manual verification is output. Only when neither condition is met will a known category be output. Specifically, the true material category of the final target to be identified can be identified from known categories such as "wood", "metal", "plastic", and "foam". Therefore, by using this material rejection output judgment condition, the situation of being forcibly classified into the wrong category can be effectively avoided. For the identification of unknown materials, "unknown material / requires manual verification" can be directly output.

[0232] Therefore, this material rejection output condition can effectively avoid the identification of unknown non-target materials, thus limiting the scope of material identification to the target material category, thereby further improving the identification accuracy and efficiency. Specifically, when the rejection rule defined by the enabled material rejection output condition is met, "unknown material" can be output, triggering manual review or safe processing parameters. In this embodiment, the material rejection output condition can be used as the content of the validation set to validate the target model architecture, resulting in a more accurate and efficient target material identification model. That is, unknown material rejection can reduce the risk of forcibly classifying materials outside the training set into known categories during the inference stage, preventing the training set from being mixed with material categories that are not the target material. It should be noted that, as mentioned above, without using the target material identification model, the methods of embodiments 1 and 2 of this invention can also achieve the corresponding material identification effect. Furthermore, the material rejection output condition can also achieve the same or similar effect as the aforementioned "preventing the risk of classifying non-target materials into known categories," which will not be elaborated further.

[0233] Based on the aforementioned cross-entropy loss and focus loss, and combined with training samples covering features such as different colors, surface roughness, processing textures, distances, angles, and lighting, a pre-trained target model architecture can be performed to enable it to have corresponding material recognition and prediction capabilities. As mentioned earlier, the pre-classifier can be based on a visual backbone such as CNN, Transformer, or a traditional feature classifier. The pre-trained target model architecture can be based on such a pre-trained model architecture. The target material recognition model can be a neural network model with high efficiency and high accuracy, formed by pre-training on a training set using the aforementioned cross-entropy loss and focus loss, and then validating it with a validation set based on the material rejection output criteria, such as a classification model based on the Transformer architecture.

[0234] In generating training samples, color perturbation, exposure perturbation, blurring, noise reduction, and local occlusion enhancement can be applied to the same material, but overly strong transformations that alter the physical texture meaning of the material should be avoided. The purpose of these enhancements is to improve the model's robustness to changes in lighting, distance, and surface defects, while maintaining the physical recognizability of the material texture. During training, batch sampling should ensure a balanced number of samples for each material type, or resampling should be used to mitigate class imbalance.

[0235] like Figures 2B-6B As shown, according to an embodiment of the present invention, after operation S224 obtains the target category attribute of the target to be identified based on the material category probability and the pre-generated material rejection output judgment condition, the method further includes:

[0236] Based on the first material category probability of the first monocular visual image of the current frame and the first fusion category probability of the second monocular visual image of the adjacent frame, a probability exponential fusion is performed using a preset fusion coefficient to generate the second fusion category probability.

[0237] Based on the second fusion category probability and the preset target tracking rules, the material identification result of the target to be identified is determined from the target category attributes.

[0238] like Figure 3 As shown, operation S307 is mainly used to realize multi-frame probability fusion and stable output process.

[0239] Among them, multi-frame probability fusion mainly involves exponentially fusing the material category probabilities of monocular visual images in consecutive frames. Specifically, it can be based on the first material category probability p_k of the first monocular visual image in the current frame (e.g., frame k) and the first fused category probability of the second monocular visual image in the previous frame (e.g., frame k-1). Determine the probability of the corresponding second fusion category in the current frame (e.g., the k-th frame). satisfy:

[0240] (35)

[0241] in, p_k is the class probability after fusion in the k-th frame, i.e., the second fused class probability; p_k is the single-frame prediction probability in the k-th frame ( =p_c), which is the probability of the first material category; α is the preset fusion coefficient, where 0 < α ≤ 1; This represents the fusion result of the previous frame, i.e., the probability of the first fusion category. Here, k is the frame number of the current frame.

[0242] Accordingly, the first material category probability p_k of the first monocular visual image in the current frame (e.g., frame k) and the first fused category probability of the second monocular visual image in the next frame (e.g., frame k+1) can be used. Determine the probability of the corresponding second fusion category in the current frame (e.g., the k-th frame). :

[0243] (36)

[0244] There are no specific restrictions, as long as the probability fusion between adjacent frames is satisfied. This allows for further integration with target tracking technology defined by preset target tracking rules, enabling accurate identification of the true material category (i.e., material identification result) of the monocular visual image in the current frame through the second fusion category probability, ensuring that the materials detected in consecutive frames belong to the same target to be identified. The preset target tracking rules can be preset rules defined by target tracking technology, and will not be elaborated upon further.

[0245] Using the aforementioned second fusion category probability, the final material category can only be confirmed as the material identification result when the maximum probability category of the same tracked target is consistent across L consecutive frames, the fusion confidence exceeds the threshold, and no unknown material rejection is triggered. Short-term missed detections can retain the previous result, while the state is immediately reset when the target's identity, center, or size changes significantly.

[0246] Therefore, by using the frame probability fusion described above, the jitter of a single frame can be effectively suppressed by accumulating consecutive frames of the same target. This greatly improves the efficiency and accuracy of material identification based on the condition of rejecting unknown materials.

[0247] Based on the above-described monocular vision-based material recognition method, this invention also provides a monocular vision-based material recognition device. The following will combine... Figure 6B The device is described in detail.

[0248] Figure 6B A schematic diagram of a material identification device 620 based on monocular vision according to an embodiment of the present invention is shown.

[0249] like Figure 6B As shown, the monocular vision-based material recognition device 620 of this embodiment includes a feature extraction module 621, a feature aggregation module 622, a probability acquisition module 623, and a material recognition module 624.

[0250] The feature extraction module 621 is used to extract visual embedding features of the associated target region based on the monocular visual image of the target to be identified and the pixel window data of the associated target region corresponding to the ranging measurement point. In one embodiment, it can be used to perform the operation S221 described above, which will not be repeated here.

[0251] The feature aggregation module 622 is used to generate visual aggregated features of visual embedding features by associating the visual quality scores of the target regions with aggregated weight data. In one embodiment, it can be used to perform the operation S222 described above, which will not be repeated here.

[0252] The probability acquisition module 623 is used to acquire the material category probability of the target to be identified relative to a preset category attribute by visually aggregating features. In one embodiment, it can be used to perform the operation S223 described above, which will not be repeated here.

[0253] The material identification module 624 is used to obtain the target category attribute of the target to be identified based on the material category probability and pre-generated material rejection output judgment conditions, and to complete the material identification. In one embodiment, it can be used to perform the operation S224 described above, which will not be repeated here.

[0254] According to embodiments of the present invention, any plurality of modules among the feature extraction module 621, feature aggregation module 622, probability acquisition module 623, and material identification module 624 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the feature extraction module 621, feature aggregation module 622, probability acquisition module 623, and material identification module 624 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any one of the three implementation methods or a suitable combination of any of them. Alternatively, at least one of the feature extraction module 621, feature aggregation module 622, probability acquisition module 623, and material identification module 624 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0255] like Figure 5 As shown, combining the aforementioned Embodiments 1 and 2, taking an RGB camera as the monocular vision sensor and a single-point ToF sensor as the ranging sensor as an example, it can be seen that the above-described method for material recognition based on monocular vision and single-point ranging in this embodiment of the invention can effectively achieve outputs including material category, classification probability, unknown class state, workpiece center pixel, target distance, distance validity, visual quality, and recognition source. Simultaneously, for platform-based or desktop-based CNC (Computer Numerical Control) controllers, different spindle speeds, feed rates, and tool parameters can be called according to the material category. Among these, unknown materials or invalid distance states only allow entry into safe mode or manual confirmation, and high-risk processing is not directly executed. Furthermore, the material category can be extended to visible surface materials such as coatings and surface treatments, and the boundaries of internal substrates or chemical compositions are not identified.

[0256] In summary, Embodiments 1 and 2 of this invention can achieve the following technical effects: Under the conditions that the target surface is clearly visible, the ToF distance is correctly associated with the surface, and the training data covers actual working conditions, this invention can improve the consistency of texture scales across distances (as in Embodiment 1), and reduce the risk of high-confidence misjudgment through unknown material rejection (as in Embodiment 2). ToF does not add material spectral information; its effect mainly comes from physical scale normalization and target association (as in Embodiment 1), and is verified using metrics such as Macro-F1, unknown class AUROC, and ECE on independent test sets (as in Embodiment 2).

[0257] Regarding the corresponding technical effects of Embodiments 1 and 2 of the present invention, the following feasibility verification methods and evaluation indicators can be further provided.

[0258] The system is divided into training, validation, and test sets based on physical workpieces to prevent data leakage caused by adjacent video frames or textures of the same workpiece crossing sets. The test set covers different distances, lighting, angles, roughness, colors, batches, and processed textures, and includes unknown materials outside the training categories.

[0259] Known categories are evaluated using precision, macro-average precision, recall, and F1 score:

[0260] Precision_c=TP_c / (TP_c+FP_c), Recall_c=TP_c / (TP_c+FN_c) (37)

[0261] Where TP_c, FP_c, and FN_c are the number of true positives, false positives, and false negatives of class c, respectively; Precision_c and Recall_c are the precision and recall of class c.

[0262] F1_c=2·Precision_c·Recall_c / (Precision_c+Recall_c) (38)

[0263] Where F1_c is the F1 score of category c; Precision_c and Recall_c have the same meaning as above.

[0264] Macro-F1=(1 / C)Σ_c F1_c (39)

[0265] Where Macro-F1 is the macro-average F1 score; C is the number of known material categories; and F1_c is the F1 score for category c.

[0266] Unknown class rejections were evaluated using AUROC, the false rejection rate for known classes, and the false rejection rate for unknown classes.

[0267] Confidence reliability is based on expected calibration error:

[0268] ECE=Σ_{b=1}^B (|S_b| / n_test)·|acc(S_b)-conf(S_b)| (40)

[0269] Where ECE is the expected calibration error; B is the number of buckets; S_b is the sample set of the b-th bucket, and |S_b| is its sample size; n_test is the total number of test samples; acc(S_b) and conf(S_b) are the actual accuracy and average confidence of the b-th bucket, respectively.

[0270] As can be seen, ablation experiments were conducted to compare various schemes, including directly scaled pure RGB, single-scale normalization, multi-scale normalization, removal of ToF projection point-ROI association, and addition of unknown material rejection. A "random shuffling of distance-material labels" method was used to check for spurious correlations. Specific performance is subject to independent test results. This invention only limits the feasible technical structure and does not pre-promise unverified numerical effects. Specifically, directly scaled pure RGB serves as the baseline; single-scale normalization verifies the effectiveness of the fixed millimeter window; multi-scale normalization verifies the gain of multi-window aggregation; removal of ToF projection point-ROI association verifies the contribution of target association to distance accuracy; and adding unknown material rejection verifies improved security.

[0271] This embodiment sets categories distinguishable by surface observation features as wood, exposed metal, engineering plastics, rubber, foam, and composite panels, and provides authentic labels through supplier records, sample batches, or other reliable methods. After target segmentation, sampling points are selected within the target, avoiding edges, shadows, and high-saturation areas. Based on the effective depth, 10 mm × 10 mm, 20 mm × 20 mm, and 40 mm × 40 mm windows are converted to pixel windows and scaled to 224 × 224. The visual backbone network aggregates multi-scale embeddings and completes classification; Time-of-Flight (ToF) distance does not enter the material feature branch.

[0272] This embodiment uses independent workpieces and distances, lighting conditions, and batches not included in the training process for testing; it uses a validation set to complete temperature scaling and rejection threshold calibration, and uses untrained categories such as glass and stone to evaluate rejection of unknown materials. During runtime, the category is output only after three consecutive frames on the same tracked target meet the confirmation criteria.

[0273] In this embodiment of the invention, for desktop CNC, flexible manufacturing, and material sorting scenarios, the surface to be identified can be located first, and the Time-of-Flight (ToF) measurement points can be associated with the target area. Then, the fixed millimeter texture window can be converted into a pixel window using the camera's axial depth. Material classification is then performed by monocular vision, and the results are output through unknown material rejection and multi-frame fusion. ToF only provides geometric scale and is not used as evidence of material category. Furthermore, the specific application scenarios in this embodiment of the invention mainly target the identification of visible surface materials and do not claim to penetrate coatings to identify internal substrates or chemical compositions.

[0274] Figure 7 The diagram illustrates an electronic device suitable for implementing a data processing method for material identification and a monocular vision-based material identification method according to embodiments of the present invention.

[0275] The electronic device provided in the embodiments of the present invention includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the above-described data processing method for material identification and the monocular vision-based material identification method.

[0276] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0277] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0278] According to an embodiment of the present invention, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0279] The present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the above-described data processing method for material identification and the monocular vision-based material identification method.

[0280] The computer-readable storage medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0281] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0282] Embodiments of the present invention also include a computer program product comprising a computer program that, when executed by a processor, implements the aforementioned data processing method for material identification and the monocular vision-based material identification method.

[0283] The computer program includes program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the methods provided in the embodiments of the present invention.

[0284] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0285] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0286] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0287] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0288] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0289] Furthermore, all actions involving the acquisition of information, signals, or data in this invention are carried out in compliance with the relevant data protection laws, regulations, and policies of the country where the invention is located, and with the authorization granted by the owner of the corresponding device.

[0290] Those skilled in the art will understand that the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0291] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of the invention is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A data processing method for material identification, characterized in that, include: For each frame of monocular visual image of the target to be identified, a set of candidate surface regions is extracted. Based on the candidate set of projection points of the ranging measurement points of the target to be identified in the image coordinate system corresponding to the monocular vision image, the candidate association score corresponding to each candidate surface region in the candidate surface region set is determined. Based on the candidate association score, at least one candidate surface region in the candidate surface region set is determined as the association target region; as well as Based on the initial ranging distance of the ranging point in the pre-calibrated ranging sensor coordinate system, corresponding to the axial effective depth in the visual sensor coordinate system, and in conjunction with pre-calibration parameters, the pixel window data of the associated target region in the image coordinate system is determined. This pixel window data is used to achieve the material recognition, including: based on the physical size information of the preset actual texture window corresponding to the associated target region. The pixel focal length parameter of the pre-calibrated parameters and the axial effective depth Determine the pixel size information of the preset actual texture window in the image coordinate system. : ； ； Among them, round(·) is for rounding to the nearest integer.

2. The method according to claim 1, characterized in that, Before extracting the candidate surface region set for each frame of the received monocular visual image of the target to be identified, the method further includes: Based on the joint calibration relationship between the visual sensor and the ranging sensor, the visual sensor acquires each frame of the original monocular image of the target to be identified, and the ranging sensor simultaneously acquires the initial ranging distance of the target to be identified based on the ranging measurement point. Each frame of the original monocular image is preprocessed to generate the monocular visual image; and the initial ranging distance is depth-processed based on the joint calibration relationship to generate the original axial depth of the corresponding ranging measurement point in the visual sensor coordinate system.

3. The method according to claim 2, characterized in that, After performing image preprocessing on each frame of the original monocular image to generate the monocular visual image; and performing depth processing on the initial ranging distance based on the joint calibration relationship to generate the original axial depth of the corresponding ranging measurement point in the visual sensor coordinate system, the method further includes: Based on the pixel projection points of the monocular visual image corresponding to the joint calibration relationship in the visual sensor coordinate system, determine the projection coordinate points of the ranging measurement point in the visual sensor coordinate system. The projection pixel coordinates of the ranging measurement point in the corresponding monocular vision image are determined based on the projection coordinates and the visual sensor intrinsic parameters of the pre-calibrated parameters.

4. The method according to claim 2, characterized in that, After performing image preprocessing on each frame of the original monocular image to generate the monocular visual image; and performing depth processing on the initial ranging distance based on the joint calibration relationship to generate the original axial depth of the corresponding ranging measurement point in the visual sensor coordinate system, the method further includes: The original axial depth is subjected to median filtering to determine the filtered axial depth of the corresponding distance measurement point. The filtered axial depth is subjected to exponential smoothing to determine the smoothed axial depth of the corresponding ranging measurement point. The validity of the smoothed axial depth of the distance measurement point corresponding to the monocular vision image is determined, and the mean of at least one historical valid depth and the smoothed axial depth determined by the validity is processed to determine the axial valid depth in the vision sensor coordinate system.

5. The method according to claim 1, characterized in that, In each frame of the received monocular visual image of the target to be identified, the candidate surface region set is extracted, including: The monocular vision image is subjected to detection and segmentation, and multiple candidate surface regions are extracted as the candidate surface region set.

6. The method according to claim 3, characterized in that, In determining the candidate association score for each candidate surface region in the candidate surface region set based on the candidate set of projection points of the ranging measurement points for the target to be identified in the image coordinate system corresponding to the monocular vision image, the following are included: Based on the candidate set of projection points corresponding to the candidate surface regions containing the projected pixel coordinates, and combined with preset normalization parameters, the candidate association score corresponding to each candidate surface region is determined.

7. The method according to claim 1, characterized in that, The step of determining at least one candidate surface region in the candidate surface region set as the association target region based on the candidate association score includes: The candidate surface regions in the candidate surface region set are sorted according to the candidate association score of each candidate surface region to generate a candidate region set; The candidate surface region with the largest candidate association score in the candidate region set is selected as the association target region.

8. A data processing apparatus for material identification, used to implement the method according to any one of claims 1-7, characterized in that, include: The region extraction module is used to extract a set of candidate surface regions for each frame of the received monocular visual image of the target to be identified. The scoring determination module is used to determine the candidate association score corresponding to each candidate surface region in the candidate surface region set based on the candidate set of projection points of the ranging measurement points for the target to be identified in the image coordinate system corresponding to the monocular vision image. The region determination module is used to determine at least one candidate surface region in the candidate surface region set as the association target region based on the candidate association score. as well as The window determination module is used to determine the pixel window data of the associated target region in the image coordinate system based on the axial effective depth in the visual sensor coordinate system corresponding to the initial ranging distance of the ranging point in the pre-calibrated ranging sensor coordinate system, and in combination with pre-calibration parameters. The pixel window data is used to realize the material recognition, including: based on the physical size information of the preset actual texture window corresponding to the associated target region. The pixel focal length parameter of the pre-calibrated parameters and the axial effective depth Determine the pixel size information of the preset actual texture window in the image coordinate system. : ； ； Among them, round(·) is for rounding to the nearest integer.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images