Part sorting method, system and equipment based on AR glasses and storage medium

Abstract

The invention discloses a part sorting method based on AR glasses, and relates to the technical field of part sorting. The method comprises the following steps: acquiring ambient light data; parameters of a data acquisition module are adjusted based on the ambient light data, the data acquisition module comprises a camera, a depth sensor and a spectrum sensor, and the parameters comprise the exposure time of the camera, the scanning precision of the depth sensor and the sampling interval of the spectrum sensor; multi-modal data of a part is acquired through a data acquisition module, the multi-modal data comprises image data of the part, point cloud data of the part and spectral data of the part, and the part is a precise electronic element; generating a target feature vector based on the multi-modal data and the weight corresponding to each modal data; and on the basis of the feature vectors of different standard parts in the standard part feature library and the target feature vector, part types are obtained, and a mechanical arm is controlled to complete part sorting according to the part types. The part sorting accuracy is improved.

Description

Technical Field

[0001] This application relates to the technical field of part sorting, and particularly relates to a part sorting method, system, device, and storage medium. Background Art

[0002] In the wave of manufacturing industry intelligence, the high efficiency and accuracy of part sorting directly support the operation speed and quality threshold of the global industrial chain. From medical equipment to the new energy industry, its technological upgrade is related to the infrastructure security of human life. Therefore, there is an urgent need for a part sorting method.

[0003] In the prior art, part types are usually recognized by capturing part images and combining template matching, and a robotic arm is driven to complete sorting.

[0004] However, since the differences between precision electronic components are often at the micron level, and strong light illumination in the factory environment easily causes surface reflection of precision electronic components, it is difficult for traditional single vision acquisition methods to accurately identify part types, resulting in low accuracy of part sorting. Summary of the Invention

[0005] This application provides a part sorting method, system, device, and storage medium based on an AR glasses, which solves the technical problem of low accuracy in sorting precision electronic components.

[0006] In the first aspect of this application, a part sorting method based on an AR glasses is provided, including: collecting ambient light data; based on the ambient light data, adjusting the parameters of a data acquisition module, the data acquisition module including a camera, a depth sensor, and a spectral sensor, the parameters including the exposure time of the camera, the scanning accuracy of the depth sensor, and the sampling interval of the spectral sensor; acquiring multimodal data of a part through the data acquisition module, the multimodal data including image data, point cloud data, and spectral data of the part, the part being a precision electronic component; generating a target feature vector based on the multimodal data and the weights corresponding to each modal data; obtaining the part type based on the feature vectors of different standard parts in a standard part feature library and the target feature vector, and controlling a robotic arm to complete part sorting according to the part type.

[0007] By adopting the above technical solution, first, the ambient light data is collected and the parameters of the data acquisition module are dynamically adjusted based on the ambient light data, which can adapt to the strong light illumination environment in the factory. Secondly, the image, point cloud and spectral data of the parts are collected by the multi-modal data acquisition module, and the target feature vector is generated by combining weights, which can comprehensively characterize the features of micron-level precision electronic components. Finally, the part type is matched based on the standard part feature library and the robotic arm is controlled for sorting, realizing the high-precision identification and sorting of precision electronic components, and effectively solving the problem of low sorting accuracy caused by the single vision acquisition method in the prior art.

[0008] Optionally, generating the target feature vector based on the multi-modal data and the weights corresponding to each modal data specifically includes: extracting the two-dimensional feature information of the part based on the image data of the part, where the two-dimensional feature information includes edge contour, texture feature, and color distribution; extracting the three-dimensional feature information of the part based on the point cloud data of the part, where the three-dimensional feature information includes surface curvature, dimension ratio, and three-dimensional shape; extracting the material feature information of the part based on the spectral data of the part, where the material feature information includes reflectivity, absorptivity, and scattering characteristics; evaluating the quality of each modal data, and adjusting the weights of each modal feature information according to each quality, where each modal feature information includes two-dimensional feature information, three-dimensional feature information, and material feature information; generating the target feature vector based on each modal feature information and the weights of each modal feature information.

[0009] By adopting the above technical solution, since two-dimensional features such as edge contour are extracted from the image data, three-dimensional features such as surface curvature are extracted from the point cloud data, material features such as reflectivity are extracted from the spectral data, and the weights are adjusted based on the quality evaluation, the feature vector can express the part features more comprehensively, thus effectively identifying the tiny differences between precision electronic components.

[0010] Optionally, extracting the three-dimensional feature information of the part based on the point cloud data of the part specifically includes: preprocessing the point cloud data to obtain preprocessed point cloud data; constructing a three-dimensional model based on the preprocessed point cloud data; obtaining the three-dimensional feature information based on the three-dimensional model.

[0011] By adopting the above technical solution, since the point cloud data is preprocessed to eliminate noise, the conversion from point cloud to model is realized through three-dimensional modeling, and the features are extracted based on the model, the accuracy of three-dimensional feature extraction is improved, providing a reliable basis for distinguishing differences.

[0012] Optionally, the quality evaluation of each modality data specifically includes: calculating the signal-to-noise ratio of each modality data; evaluating the discriminability of each modality data based on a preset part sample library, where the discriminability is the difference degree between the modality data of the part and the corresponding standard data in the part sample library; detecting the integrity of each modality data, where the integrity is the ratio of the item quantity of the actually obtained modality data of each modality to the total number of item quantities of the modality standard data; and determining the data quality of each modality data based on the signal-to-noise ratio, discriminability, and integrity.

[0013] By adopting the above technical solution, since the clarity of the data is evaluated by calculating the signal-to-noise ratio, the discrimination ability of the data is evaluated by comparing with the sample library, the sufficiency of the data is evaluated by integrity detection, and the data quality is comprehensively determined based on these three dimensions, a comprehensive evaluation of the data quality is achieved, thereby providing a reliable basis for feature weight optimization.

[0014] Optionally, based on the feature vectors of different standard parts in the standard part feature library and the target feature vector, obtaining the part type specifically includes: classifying the standard parts based on the feature vectors of different standard parts in the standard part feature library to obtain multiple feature vector sets, each feature vector set corresponding to a major part category, and each major part category including at least one minor part category, where the minor part category is the part type; determining the candidate feature vector set to which the target feature vector belongs from the multiple feature vector sets; calculating the similarity between each feature vector in the candidate feature vector set and the target feature vector; and selecting the minor part category corresponding to the target candidate feature vector with the highest similarity as the part type.

[0015] By adopting the above technical solution, since the standard part feature vectors are first grouped by major category to form feature vector sets, then the candidate set to which the target feature vector belongs is determined, and finally similarity calculation and matching are performed in the candidate set, a hierarchical classification strategy is constructed, thereby improving the classification efficiency while ensuring the recognition accuracy.

[0016] Optionally, selecting the minor part category corresponding to the candidate feature vector with the highest similarity as the part type specifically includes: if there are multiple target candidate feature vectors with the highest similarity, calculating the comprehensive weights of the target candidate feature vectors based on the historical matching accuracies of the target candidate feature vectors and the ambient light data, where the historical matching accuracy is the statistical result of the correct and incorrect sorting of the standard parts under different ambient light data; and selecting the minor part category corresponding to the target candidate feature vector with the highest comprehensive weight as the part type.

[0017] By adopting the above technical solution, when there are multiple candidates with the highest similarity, the empirical data of historical matching accuracy is considered, and combined with the current ambient light conditions for comprehensive weight calculation. Therefore, historical experience and real-time environmental factors are fused, further improving the reliability of the classification result.

[0018] Optionally, it is characterized in that the control robotic arm completes part sorting according to the part type, specifically including: generating a grasping strategy according to the geometric features of the part, the grasping strategy including a grasping position and a grasping angle; planning a motion path of the robotic arm based on the target position information corresponding to the part type and the grasping strategy; controlling the robotic arm to perform part sorting according to the grasping strategy and the motion path.

[0019] By adopting the above technical solution, since a grasping strategy including a grasping position and an angle is generated based on the geometric features of the part, and the motion path is planned in combination with the target position, and then the sorting action is performed according to the plan, the intelligent planning of the robotic arm movement is realized, thus ensuring the safety and reliability of the sorting process of precision electronic components.

[0020] In the second aspect of the present application, a part sorting system based on an AR glasses is provided, specifically including: An ambient light perception module for collecting ambient light data; An adaptive parameter regulation module for adjusting the parameters of the data acquisition module based on the ambient light data, the data acquisition module including a camera, a depth sensor, and a spectral sensor, and the parameters including the exposure time of the camera, the scanning accuracy of the depth sensor, and the sampling interval of the spectral sensor; A multi-modal synchronous acquisition module for acquiring multi-modal data of the part through the data acquisition module, the multi-modal data including image data of the part, point cloud data of the part, and spectral data of the part, and the part being a precision electronic component; A feature fusion engine module for generating a target feature vector based on the multi-modal data and the weights corresponding to each modal data; An intelligent sorting decision module for obtaining the part type based on the feature vectors of different standard parts in the standard part feature library and the target feature vector, and controlling the robotic arm to complete part sorting according to the part type.

[0021] By adopting the above technical solution, since the ambient light data is collected and the parameters of each sensor are adjusted accordingly, the interference of strong light illumination in the factory can be overcome. Secondly, through the fusion acquisition of multi-modal data, the image, point cloud, and spectral data of the part are obtained, thus providing richer feature information. Finally, feature fusion is realized based on weight optimization, so the problem of low sorting accuracy of precision electronic components can be effectively solved.

[0022] In a third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface, and a network interface. The memory is used to store instructions. Both the user interface and the network interface are used to communicate with other devices. The processor is used to execute the instructions stored in the memory, so that the electronic device executes the method described in any one of the above.

[0023] In a fourth aspect of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions, and when the instructions are executed, the method described in any one of the above is executed. Description of the Drawings

[0024] Figure 1 is a schematic diagram of the architecture of a part sorting system based on an AR glasses disclosed in an embodiment of the present application; Figure 2 is a schematic flowchart of a part sorting method based on an AR glasses disclosed in an embodiment of the present application; Figure 3 is Figure 2 a schematic sub-step flowchart of step S104; Figure 4 is Figure 3 a schematic sub-step flowchart of step S202; Figure 5 is Figure 3 a schematic sub-step flowchart of step S204; Figure 6 is Figure 2 a schematic sub-step flowchart of step S105; Figure 7 is Figure 6 a schematic sub-step flowchart of step S504; Figure 8 is Figure 2 a schematic sub-step flowchart of step S105; Figure 9 is a schematic module diagram of a part sorting system based on an AR glasses provided by an embodiment of the present application; Figure 10 is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application.

[0025] Description of the reference numerals: 21, ambient light perception module; 22, adaptive parameter regulation module; 23, multi-modal synchronous acquisition module; 24, feature fusion engine module; 25, intelligent sorting decision module; 901, processor; 902, communication bus; 903, user interface; 904, network interface; 905, memory. Detailed Embodiments

[0026] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments.

[0027] In the description of the embodiments of this application, words such as "for example" or "for instance" are used to give examples, illustrations or explanations. Any embodiment or design solution described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0028] In the description of the embodiments of this application, the term "a plurality of" means two or more. For example, a plurality of systems means two or more systems, and a plurality of screen terminals means two or more screen terminals. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. The terms "comprise", "include", "have" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0029] Figure 1 An exemplary system architecture 10 of a part sorting system based on AR glasses is shown.

[0030] As Figure 1 shown, the system architecture 10 may include an AR glasses 11, a network 12, and a robotic arm 13. The network 12 is used to provide a medium for communication links between the terminal device AR glasses 11 and the robotic arm 13. The network 12 may include various wireless communication methods, such as long- and short-distance wireless communication, wide-area communication, etc.

[0031] The user can use the AR glasses 11 to interact with the robotic arm 13 through the network 12 to receive or send data, etc.

[0032] The AR glasses 11 are used to acquire data, process data, and issue instructions.

[0033] The robotic arm 13 can receive the instructions transmitted by the AR glasses 11 for operation processing and can feedback the processing results to the AR glasses 11.

[0034] It should be understood, Figure 1The numbers of the AR glasses 11, network 12, and robotic arm 13 in it are merely illustrative. According to the implementation requirements, any number of AR glasses 11, network 12, and robotic arm 13 can be provided.

[0035] This embodiment discloses a part sorting method based on AR glasses. Referring to Figure 2 , Figure 2 FIG. is a schematic flowchart of a part sorting method based on AR glasses provided by an embodiment of the present application, including steps S101 to S105: S101: Collect ambient light data.

[0036] Among them, the ambient light data refers to the illumination parameter information in the working scene, including characteristics such as illumination intensity, color temperature, illuminance uniformity, and light source direction. The working scene includes the area for part recognition within the field of view of the AR glasses, such as a desktop workbench, an assembly line, a storage shelf, etc.

[0037] Specifically, first, it is necessary to activate the light sensor of the AR glasses, then calibrate the sensor parameters and measurement range, and then continuously collect the ambient light data within the user's field of view.

[0038] Optionally, a real-time acquisition method is used for measurement. By starting the light sensor of the glasses, setting appropriate sampling frequencies and accuracy parameters, continuously obtaining ambient light parameters and recording the corresponding timestamp information, a light data sequence is formed.

[0039] Optionally, a scene recognition method is used for measurement. By analyzing the type of scene where the user is located, dynamically adjusting the measurement parameter configuration, an adaptive acquisition strategy is realized to ensure that the data quality meets the requirements.

[0040] It can be understood that other acquisition methods such as multi-sensor fusion and intelligent triggering can also be used to obtain the ambient light data of the AR glasses, which are not limited here.

[0041] S102: Based on the ambient light data, adjust the parameters of the data acquisition module. The data acquisition module includes a camera, a depth sensor, and a spectral sensor. The parameters include the exposure time of the camera, the scanning accuracy of the depth sensor, and the sampling interval of the spectral sensor.

[0042] Among them, the data acquisition module refers to various sensor components in the AR glasses used to sense the environment. The various sensor components include a camera, a depth sensor, and a spectral sensor.

[0043] Specifically, first analyze the characteristics of the ambient light data, then calculate the optimal parameter values of each sensor, and then update the parameter configuration of the sensor.

[0044] Optionally, by establishing the correspondence between the ambient light intensity and the sensor parameters, the appropriate values of the camera exposure time, depth scanning accuracy, and spectral sampling interval are calculated in real time to ensure the data acquisition quality.

[0045] Optionally, by evaluating the sensor data quality in real time and dynamically fine-tuning the parameter configurations, the balance between the acquisition effect and the system resource consumption is achieved to realize the optimal performance.

[0046] S103: Obtain the multimodal data of the part through the data acquisition module. The multimodal data includes the image data of the part, the point cloud data of the part, and the spectral data of the part, and the part is a precision electronic component.

[0047] Among them, the multimodal data refers to obtaining the image data of the part, the point cloud data of the part, and the spectral data of the part through the data acquisition module. The part is a precision electronic component, such as a chip, a capacitor, a resistor, a transistor, etc.

[0048] Specifically, first ensure that the part is within the field of view of the acquisition device, then synchronously start each sensor for data acquisition, and then perform time alignment and storage on the obtained multimodal data.

[0049] Optionally, by coordinating the acquisition timings of the camera, depth sensor, and spectral sensor, the image, point cloud, and spectral data of the part are synchronously acquired to ensure the time consistency of the multimodal data.

[0050] Optionally, by dividing the part into regions and respectively acquiring the multimodal data of each region, the acquisition accuracy and spatial resolution are improved to realize the complete part feature expression.

[0051] It can be understood that other acquisition methods such as multi-angle acquisition and multi-scale acquisition can also be adopted to realize the acquisition of the part multimodal data, which is not limited here.

[0052] S104: Generate the target feature vector based on the multimodal data and the weights corresponding to each modal data.

[0053] Specifically, first preprocess each modal data to unify the data in different ranges into the same numerical interval. Then extract the key numerical features from each modal data to form a preliminary digital representation. Next, calculate the reliability of each modal feature and adjust the weights during fusion according to the reliability. After that, add the modal features weighted by the weights to obtain the preliminary fusion feature. Finally, perform simplification and standardization processing on the fusion feature to obtain the final target feature vector.

[0054] Refer to Figure 3 , Figure 3 which is provided by the embodiment of the present application Figure 2Schematic diagram of a sub-step process of step S104, including steps S201 to S204. The above steps are as follows: S201: Based on the image data of the part, extract the two-dimensional feature information of the part. The two-dimensional feature information includes edge contour, texture feature, and color distribution.

[0055] Among them, the two-dimensional feature information includes edge contour, texture feature, and color distribution. The edge contour refers to the outer boundary and internal structure lines of the part, such as the outer shape of the part, internal holes, etc. The texture feature represents the detailed structural features on the surface of the part, such as grinding lines on the surface of a metal part, injection molding texture on the surface of a plastic part, etc. The color distribution refers to the color information of different regions on the surface of the part, such as color changes on the surface of the part, material reflection, etc.

[0056] Specifically, first preprocess the part image, including image enhancement, noise removal, and size unification. Then extract the edge contour information respectively, identify the main boundary lines and internal structure lines of the part. Next, analyze the texture information on the surface of the part to obtain the change characteristics of the surface structure. After that, statistically analyze the color information of each region of the part, establish a color histogram and a spatial distribution map. Finally, integrate the edge, texture, and color features to form complete two-dimensional feature information.

[0057] Optionally, first divide the part image into multiple regions, then extract the edge and texture information for each region respectively. Next, calculate the color statistical features of each region, and then merge the feature information of adjacent regions. Finally, integrate the features of all regions to form two-dimensional feature information.

[0058] Optionally, first perform edge detection on the entire image to obtain contour information, then analyze the change of texture features in the overall image. Next, establish a global color distribution model, and then organize the three features according to their position relationships. Finally, generate unified two-dimensional feature information.

[0059] It can be understood that other extraction methods such as multi-scale feature extraction and hierarchical feature extraction can also be used to obtain the two-dimensional feature information of the part, which is not limited here.

[0060] S202: Based on the point cloud data of the part, extract the three-dimensional feature information of the part. The three-dimensional feature information includes surface curvature, size ratio, and three-dimensional shape.

[0061] Among them, the point cloud data is a set of spatial data composed of a large number of points. Each point contains spatial position coordinates. The three-dimensional feature information is a set of part spatial features extracted from these point cloud data, which contains the main structural information of the part. The surface curvature reflects the arc change characteristics of the part surface and is calculated through the distribution curve of local points, which can represent features such as planes, convex surfaces, and concave surfaces. The size ratio reflects the size relationship between different parts of the part. After obtaining the basic dimensions by measuring the spatial distribution range of the point cloud, the ratio is calculated. The three-dimensional shape describes the overall spatial structure characteristics of the part, which can be a single basic geometric body such as a cylinder or a cube, or a combination of multiple basic geometric bodies.

[0062] Specifically, first, perform data preprocessing on the original point cloud to remove the miscellaneous points and data jitter generated during the measurement process, making the distribution of points more uniform. Then, calculate the curvature information of the local area on the processed point cloud to find the feature areas such as planes, protrusions, and depressions. Next, measure the basic dimension data of the part through the spatial distribution of the point cloud to determine the length, width, and height information of each part. After that, analyze the clustering relationship of the points to identify the spatial shape characteristics of the part. Finally, combine the surface curvature, size ratio, and three-dimensional shape characteristics into complete three-dimensional feature information.

[0063] Optionally, first divide the overall point cloud into several local point sets, then calculate the surface curvature characteristics of each local point set respectively, then measure the size ratio parameters of each local point set, and then infer the three-dimensional shape of this area based on the local characteristics. Finally, combine all the local characteristics into complete three-dimensional feature information.

[0064] Optionally, first analyze the surface curvature distribution of the entire point cloud, then calculate the overall size ratio of the part, then identify the three-dimensional shape according to the distribution law of the points, and then associate these features according to the spatial structure. Finally, generate complete three-dimensional feature information.

[0065] Refer to Figure 4 , Figure 4 is a schematic diagram of a sub-step process of step S202 provided in the embodiment of the present application, including steps S301 to S303. The above steps are as follows: Figure 3 S301: Perform preprocessing on the point cloud data to obtain preprocessed point cloud data.

[0066] Among them, the preprocessed point cloud data refers to the set of points after filtering, noise reduction, and homogenization processing. Specifically, the point cloud data is first subjected to denoising to remove noise points caused by equipment errors and environmental interference, and then the redundant points generated by repeated sampling are cleared. The point cloud is then subjected to density homogenization to make the distribution of points more regular, and then data smoothing is performed to reduce local fluctuations. Finally, the processed point cloud is spatially aligned and scale-normalized to obtain the preprocessed point cloud data.

[0067] Optionally, first calculate the neighborhood statistical features of each point, then identify abnormal points based on the statistical features, then remove noise points and outliers, and then uniformly sample the remaining point cloud to finally obtain preprocessed point cloud data.

[0068] Optionally, the space is first divided into regular grids, and then the distribution characteristics of the points in each grid are calculated, and then the representative points in each grid are retained, and then the grid points are locally smoothed, and finally the preprocessed point cloud data is generated.

[0069] S302: Construct a three-dimensional model based on the pre-processed point cloud data.

[0070] Among them, the three-dimensional model refers to the digital surface model of the part reconstructed from point cloud data, including mesh patches, surfaces or solids.

[0071] Specifically, we first estimate the normal vector of the preprocessed point cloud and calculate the directional characteristics of the local surface of each point. Then, we build the topological connection between the points to form a preliminary grid structure. The grid is then optimized and repaired to fill in the missing areas and smooth the irregular parts. After that, we perform surface fitting based on the grid model to generate a smooth surface model. Finally, we simplify and optimize the model to obtain a complete three-dimensional model.

[0072] Optionally, the normal vector information of the point cloud is calculated first, and then a triangular mesh is generated using an algorithm such as Poisson reconstruction, followed by repairing damage and defects in the mesh, and then optimizing and simplifying the mesh to finally obtain a complete three-dimensional model.

[0073] Optionally, first identify basic geometric features in the point cloud, then fit basic surfaces such as planes and cylinders, then construct transition surfaces between features, then connect each feature surface into a complete model, and finally generate a complete three-dimensional model.

[0074] S303: Acquire three-dimensional feature information based on the three-dimensional model.

[0075] Among them, the three-dimensional feature information includes surface curvature, size ratio and three-dimensional shape, which reflect the key geometric properties of the parts.

[0076] Specifically, first, divide the three-dimensional model into characteristic regions to identify surface regions with different properties, then calculate the curvature distribution of the model surface to obtain information on the bending changes of the curved surface, then measure the main dimensional parameters of the model to establish the dimensional proportional relationships of each part, then analyze the spatial structure characteristics of the model to extract basic three-dimensional shape information, and finally integrate various types of features to obtain complete three-dimensional feature information.

[0077] Optionally, first divide the three-dimensional model into multiple local regions, then calculate the surface curvature characteristics of each region, then measure the dimensional proportions of the local regions, then identify the three-dimensional shape of each region, and finally combine all regional features to form complete three-dimensional feature information.

[0078] Optionally, first analyze the surface curvature distribution of the entire model, then calculate the overall dimensional proportion of the model, then extract the basic three-dimensional shape of the model, then correlate these features according to the spatial structure, and finally generate complete three-dimensional feature information.

[0079] S203: Based on the spectral data of the part, extract the material feature information of the part. The material feature information includes reflectivity, absorptivity, and scattering characteristics.

[0080] Among them, the spectral data is the optical response data of the part surface collected by a spectral measurement device, which records the interaction characteristics between the part surface and light under illumination of different wavelengths. The material feature information is a set of material optical characteristics extracted from the spectral data. Reflectivity represents the reflection ability of the surface to incident light, absorptivity represents the degree of light energy absorption by the material, and scattering characteristics describe the scattering distribution law of light on the material surface.

[0081] Specifically, first preprocess the spectral data to eliminate measurement noise and equipment errors, then analyze the spectral response curves at different wavelengths to extract characteristic band information, then calculate the reflectivity and absorptivity parameters of the material, then analyze the scattering distribution characteristics of light, and finally combine these optical characteristics into complete material feature information.

[0082] Optionally, first segment the spectral data by wavelength, then calculate the reflectivity and absorptivity of each band respectively, then analyze the scattering characteristics of each band, then integrate the characteristics of each band, and finally obtain complete material feature information.

[0083] Optionally, first analyze the change trend of the complete spectral curve, then calculate the overall reflection and absorption characteristics, then extract the scattering law of the material, then correlate these features, and finally generate complete material feature information.

[0084] S204: Evaluate the quality of each modality data, and adjust the weights of the feature information of each modality according to the quality. The feature information of each modality includes two-dimensional feature information, three-dimensional feature information, and material feature information.

[0085] Among them, the modality data includes image data, point cloud data, and spectral data. The data quality reflects the integrity and reliability of the collected data. The weight represents the importance of different feature information in subsequent processing. The two-dimensional feature information describes the planar contour features of the part, the three-dimensional feature information describes the spatial shape features of the part, and the material feature information describes the material optical characteristics of the part.

[0086] Specifically, first evaluate the clarity and integrity of the image data, calculate the reliability score of the two-dimensional features, then evaluate the density and uniformity of the point cloud data to obtain the quality index of the three-dimensional features, then evaluate the signal-to-noise ratio and stability of the spectral data to obtain the credibility of the material features, and then determine the weights of the corresponding feature information according to each quality score.

[0087] Optionally, first normalize the quality scores of each modality data, then set the feature weights according to the scoring ratio, then perform weighted combination on the feature information, then verify the rationality of the weight allocation, and finally obtain the adjusted feature weights.

[0088] Optionally, first establish the mapping relationship between data quality and feature weights, then dynamically calculate the weights according to the quality evaluation results, then perform weight optimization adjustment, then balance the importance of each feature, and finally generate the optimal feature weights.

[0089] Refer to Figure 5 , Figure 5 is a schematic diagram of a sub-step process of step S204 provided by an embodiment of the present application, including steps S401 to S404. The above steps are as follows: Figure 3 S401: Calculate the signal-to-noise ratio of each modality data. S401: Calculate the signal-to-noise ratio of each modality data.

[0090] Among them, the signal-to-noise ratio refers to the ratio of the effective signal strength to the noise strength.

[0091] Specifically, first estimate the signal and noise energy in the image data, calculate the image signal-to-noise ratio, then analyze the point position error and distribution noise of the point cloud data to obtain the point cloud signal-to-noise ratio, then evaluate the signal fluctuation and background noise of the spectral data to obtain the spectral signal-to-noise ratio, and then perform standardization processing on the signal-to-noise ratios of each modality, and finally obtain a unified signal-to-noise ratio Optionally, first divide the data into multiple local regions, then calculate the signal and noise intensities of each region respectively, then obtain the local signal-to-noise ratio, then calculate the average value of all regions, and finally obtain the overall signal-to-noise ratio.

[0092] Optionally, first extract the overall signal characteristics of the data, then estimate the noise level in the data, then calculate the energy ratio of the signal to the noise, then perform numerical correction, and finally generate the signal-to-noise ratio.

[0093] S402: Evaluate the distinguishability of each modal data based on a preset part sample library, where the distinguishability is the degree of difference between each modal data of the part and the corresponding standard data in the part sample library.

[0094] Among them, the part sample library contains a set of standard data for different types of parts. The modal data includes image data, point cloud data, and spectral data. The distinguishability reflects the degree of difference between the data of the part to be measured and the standard data.

[0095] Specifically, first match the standard data of the same type of part from the sample library, then calculate the degree of difference between the image data and the standard image, then measure the deviation degree between the point cloud data and the standard point cloud, then analyze the change amount between the spectral data and the standard spectrum, and finally comprehensively obtain the distinguishability of each modal data.

[0096] Optionally, first extract the features of the data to be measured and the standard data, then calculate the distance between the features, then normalize the difference value, then count the distinguishability degree of each feature, and finally obtain the overall distinguishability.

[0097] Optionally, first align the data to be measured with the standard data, then directly calculate the difference between the data, then quantify the size of the difference, then perform difference statistics, and finally obtain the data distinguishability.

[0098] S403: Detect the integrity of each modal data, where the integrity is the ratio of the number of items of the actually obtained each modal data to the total number of items of the each modal standard data.

[0099] Among them, the integrity refers to the completeness of data acquisition. For image data, the integrity represents the ratio of the number of feature points obtained to the total number of standard feature points. For point cloud data, the integrity represents the ratio of the actually collected number of points to the ideal coverage number of points. For spectral data, the integrity represents the ratio of the number of effective wavebands to the total number of standard wavebands.

[0100] Specifically, first count the ratio of the number of effective pixels of the image data to the total number of standard pixels, then calculate the ratio of the number of effective points of the point cloud data to the number of standard points, then measure the ratio of the number of effective wavebands of the spectral data to the total number of standard wavebands, then normalize the integrity values of each modality, and finally obtain the data integrity.

[0101] Optionally, first regionally segment the data, then count the data item ratio of each region, then merge the regional integrity values, then balance the weights of each region, and finally obtain the overall integrity.

[0102] S404: Determine the data quality of each modality data based on signal-to-noise ratio, discrimination, and integrity.

[0103] Specifically, first obtain the signal-to-noise ratio of each modality data, then read the discrimination of each modality data, then import the integrity of each modality data, then determine the data quality by synthesizing the three indicators, and finally obtain the data quality of each modality data.

[0104] Optionally, by setting fixed weight coefficients for signal-to-noise ratio, discrimination, and integrity, calculate the weighted average score to obtain the data quality of each modality data.

[0105] Optionally, first establish a hierarchical evaluation system, then calculate the indicator scores layer by layer, then summarize the results of each layer, then balance the indicator impacts, and finally obtain the data quality of each modality data.

[0106] S205: Generate a target feature vector based on each modality feature information and the weights of each modality feature information.

[0107] Specifically, first obtain the two-dimensional feature information and its weight, then read the three-dimensional feature information and its weight, then import the material feature information and its weight, then fuse each feature information according to the weight, and finally generate a target feature vector.

[0108] Optionally, first standardize each feature information, then perform weighting according to the weight, then splice the feature components, then verify the vector dimension, and finally obtain a feature vector.

[0109] Optionally, first construct a feature mapping model, then input the weighted feature information, then calculate the mapping result, then optimize the vector representation, and finally obtain a feature vector.

[0110] S105: Based on the feature vectors of different standard parts in the standard part feature library and the target feature vector, obtain the part type, and control the robotic arm to complete part sorting according to the part type.

[0111] Among them, the standard parts include mechanical parts such as screws, nuts, and washers. The standard part feature library is a data set of the feature vectors of these parts. The target feature vector is composed of two-dimensional feature information, three-dimensional feature information, and material feature information combined according to the weight. The feature vector is a mathematical expression form of the part features.

[0112] Specifically, first obtain the set of feature vectors in the standard part feature library, then calculate the distance between the target feature vector and the standard feature vector, then determine the part type according to the minimum distance, then plan the sorting path of the robotic arm, and finally control the robotic arm to complete part sorting.

[0113] Optionally, first calculate the Euclidean distance or cosine similarity between feature vectors, then determine the part type according to the distance threshold, then generate the grasping position and placement position of the robotic arm, then drive the robotic arm to complete the movement, and finally place the part at the target position.

[0114] Optionally, first extract the shape pattern and material pattern of the feature vector, then identify the part type through a classifier, then determine the sorting order and target area, then control the robotic arm to perform the handling operation, and finally complete the classified placement of the parts.

[0115] Referring to Figure 6 , Figure 6 is a sub-step process schematic diagram of step S105 provided by an embodiment of the present application, including steps S501 to S504, and the above steps are as follows: Figure 2 S501: Classify standard parts based on the feature vectors of different standard parts in the standard part feature library to obtain multiple feature vector sets. Each feature vector set corresponds to a major part category, and each major part category includes at least one minor part category. The minor part category is the part type.

[0116] Among them, the standard parts are precision electronic components, including resistor devices, capacitor devices, and inductor devices. The standard part feature library is a collection library of the feature vectors of these devices. The feature vector set is a group of device features with similar features. The major part category is a set of categories of basic devices, and the minor part category is the specific model of the device. For example, under the major category of resistor devices, there are minor categories such as surface mount resistor models and through-hole resistor models.

[0117] Specifically, first obtain all the feature vectors in the standard part feature library, then perform major category clustering according to the similarity of the feature vectors, then perform minor category division within each major category, then establish the hierarchical relationship between the major category and the minor category, and finally generate multiple feature vector sets.

[0118] Optionally, first calculate the distance value between feature vectors, then obtain the major component category using a clustering system, then determine the minor component category through feature differences, then establish a classification index table, and finally obtain multiple feature vector sets.

[0119] Optionally, use the feature analysis method for classification. First, extract the numerical features of the feature vector, then divide the major component category according to the feature correlation degree, then determine the minor component category based on the feature details, then generate a category structure tree, and finally form multiple feature vector sets.

[0120] S502: Determine the candidate feature vector set to which the target feature vector belongs from multiple feature vector sets.

[0121] Among them, the feature vector set is a feature data group of the same type of electronic components, the target feature vector is the feature representation of the component to be classified, and the candidate feature vector set is a feature group similar to the target feature vector.

[0122] Specifically, first obtain the numerical representation of the target feature vector, then calculate the distance values between the target feature vector and each feature vector set, then sort according to the distance size, then select the feature vector set with the smallest distance, and finally determine the candidate feature vector set. The distance is the numerical difference between feature vectors.

[0123] S503: Calculate the similarity between each feature vector in the candidate feature vector set and the target feature vector.

[0124] Specifically, first read all the feature vectors in the candidate feature vector set, then obtain the numerical representation of the target feature vector, and then calculate the similarity values of each pair of feature vectors.

[0125] Optionally, first extract the component values of the feature vector, then calculate the inner product and length of the vector, then use the cosine calculation formula, and then perform numerical normalization, and finally obtain the similarity value.

[0126] Optionally, first obtain each component of the feature vector, then calculate the sum of the squares of the component differences, then convert it to a similarity measure, and then perform proportional scaling, and finally obtain the similarity value.

[0127] S504: Select the part subclass corresponding to the target candidate feature vector with the highest similarity as the part type.

[0128] Specifically, first obtain the similarity values of all feature vectors, then sort and compare the similarities, then determine the feature vector with the highest similarity, then find the part subclass corresponding to this feature vector, and finally output the part type.

[0129] Optionally, first select the feature vector with the maximum similarity, then query the type index of the feature vector, then obtain the corresponding part subclass, and then verify the accuracy of the type, and finally determine the part type.

[0130] Optionally, first set the threshold standard of the similarity, then screen the feature vectors that meet the threshold, then compare the similarities of the screening results, then select the optimal matching item, and finally obtain the part type.

[0131] Refer to Figure 7 , Figure 7 is a sub-step flow schematic diagram of step S504 provided by an embodiment of the present application, including steps S601 to S602. The above steps are as follows: Figure 6 S601: If there are multiple target candidate feature vectors with the highest similarity, calculate the comprehensive weight of each target candidate feature vector based on the historical matching accuracy rate of each target candidate feature vector and the ambient light data. The historical matching accuracy rate is the statistical result of the correct and incorrect sorting of the standard parts under different ambient light data.

[0132] Among them, the historical matching accuracy rate is the statistical data of the sorting result, and the ambient light data is the value of the current light intensity. For example, the historical sorting correct rate of a certain type of resistor under 200 lux light is 95%. Specifically, first obtain multiple feature vectors with the highest similarity, then read the historical accuracy rate of each feature vector, then collect the current ambient light data, and then combine the accuracy rate and the light conditions, and finally calculate the comprehensive weight.

[0133] Optionally, first set the weight coefficients of the accuracy rate and the light, then standardize the data, then calculate the weighted sum value, then perform weight normalization, and finally obtain the comprehensive weight.

[0134] Optionally, first establish a probability model of the accuracy rate and the light, then calculate the conditional probability of each feature vector, then perform probability correction, then convert it into a weight value, and finally obtain the comprehensive weight.

[0135] S602: Select the part subclass corresponding to the target candidate feature vector with the highest comprehensive weight as the part type.

[0136] Specifically, first obtain the comprehensive weights of all target candidate feature vectors, then sort the weight values, then select the feature vector with the highest weight, then find the corresponding part subclass, and finally determine the part type.

[0137] Optionally, first compare the weight values of each feature vector, then determine the item with the largest weight, then obtain the corresponding type information, then perform result verification, and finally obtain the part type.

[0138] Optionally, first set the threshold range of the weight, then screen the feature vectors that meet the requirements, then select the optimal item from the screening results, then confirm the type attribute, and finally determine the part type.

[0139] Refer to Figure 8 , Figure 8 is a sub-step flow schematic diagram of step S105 provided by the embodiment of the present application, including steps S701 to S703. The above steps are as follows: Figure 2 S701: Generate a grasping strategy according to the geometric features of the part. The grasping strategy includes the grasping position and the grasping angle.

[0140] ​Among them, geometric features are the set of shape parameters of the part, including dimension data, contour data, and surface data. The grasping strategy is the action plan of the robotic arm, the grasping position is the acting coordinate point of the gripper, and the grasping angle is the rotation direction of the gripper. For example, for a rectangular resistor, it can be grasped at the midpoint of the long side at a 90-degree angle.

[0141] Specifically, first obtain the geometric feature parameters of the part, then analyze the shape characteristics of the part, then determine the optimal grasping position coordinates, then calculate the appropriate grasping angle value, and finally generate the grasping strategy.

[0142] Optionally, first extract the edge point set of the part, then calculate the feasible grasping point group, then evaluate the stability score of each point, then select the optimal grasping scheme, and finally output the position and angle.

[0143] Optionally, first establish a strategy template library for the part shape, then match the shape type of the current part, then obtain the strategy parameters in the template, then optimize the parameters, and finally determine the grasping scheme.

[0144] S702: Plan the motion path of the robotic arm based on the target position information and grasping strategy corresponding to the part type.

[0145] Specifically, first obtain the target placement area of the part type, then read the position angle of the grasping strategy, then calculate the starting and ending points of the robotic arm, then generate the obstacle avoidance path nodes, and finally plan the motion path.

[0146] Optionally, use the node planning method for planning. First, determine the key node group of the path, then calculate the safe path between the nodes, then optimize the node connection order, then generate a smooth motion curve, and finally obtain the motion path.

[0147] Optionally, first divide the safe area of the working space, then find a feasible path within the safe area, then optimize the path length, then add speed and acceleration constraints, and finally obtain the motion path.

[0148] S703: Control the robotic arm to perform part sorting according to the grasping strategy and motion path.

[0149] Among them, the robotic arm is the machine device that performs the sorting task, including the mechanical body, the gripper component, and the controller.

[0150] Specifically, first initialize the control system of the robotic arm, then move to the position specified by the grasping strategy, then perform the grasping action at the set angle, then move along the planned path, and perform part sorting.

[0151] Optionally, first obtain the node coordinates in the path, then execute the position instructions in sequence, then control the opening and closing actions of the gripper, and then adjust the motion speed parameters to perform part sorting.

[0152] Optionally, first generate a continuous motion trajectory, then calculate the angular changes of each joint, then execute the servo control instructions, and then monitor the motion state feedback to perform part sorting.

[0153] Reference Figure 9 , this application also provides a part sorting system 20 based on AR glasses. Specifically, it includes: An ambient light sensing module 21 for collecting ambient light data; An adaptive parameter regulation module 22 for adjusting the parameters of the data acquisition module based on the ambient light data. The data acquisition module includes a camera, a depth sensor, and a spectral sensor, and the parameters include the exposure time of the camera, the scanning accuracy of the depth sensor, and the sampling interval of the spectral sensor; A multi-modal synchronous acquisition module 23 for obtaining multi-modal data of the part through the data acquisition module. The multi-modal data includes image data of the part, point cloud data of the part, and spectral data of the part, and the part is a precision electronic component; A feature fusion engine module 24 for generating a target feature vector based on the multi-modal data and the weights corresponding to each modal data; An intelligent sorting decision module 25 for obtaining the part type based on the feature vectors of different standard parts in the standard part feature library and the target feature vector, and controlling the robotic arm to complete part sorting according to the part type.

[0154] Optionally, the feature fusion engine module 24 is further configured to extract two-dimensional feature information of the part based on the image data of the part. The two-dimensional feature information includes edge contour, texture feature, and color distribution; extract three-dimensional feature information of the part based on the point cloud data of the part. The three-dimensional feature information includes surface curvature, size ratio, and three-dimensional shape; extract material feature information of the part based on the spectral data of the part. The material feature information includes reflectivity, absorptivity, and scattering characteristics; evaluate the quality of each modal data, and adjust the weights of each modal feature information according to each quality. The modal feature information includes two-dimensional feature information, three-dimensional feature information, and material feature information; generate a target feature vector based on the modal feature information and the weights of the modal feature information.

[0155] Optionally, the feature fusion engine module 24 is further configured to preprocess the point cloud data to obtain preprocessed point cloud data; construct a three-dimensional model based on the preprocessed point cloud data; and obtain the three-dimensional feature information based on the three-dimensional model.

[0156] Optionally, the feature fusion engine module 24 is further configured to calculate the signal-to-noise ratio of each modality data. Evaluate the discriminability of each modality data based on a preset part sample library, where the discriminability is the difference between the modality data of the part and the corresponding standard data in the part sample library; detect the integrity of each modality data, where the integrity is the ratio of the number of items of the actually obtained modality data to the total number of items of the modality standard data; and determine the data quality of each modality data based on the signal-to-noise ratio, discriminability, and integrity.

[0157] Optionally, the intelligent sorting decision module 25 is further configured to classify the standard parts based on the feature vectors of different standard parts in the standard part feature library to obtain multiple feature vector sets, each feature vector set corresponding to a major part category, and each major part category including at least one minor part category, where the minor part category is the part type; determine the candidate feature vector set to which the target feature vector belongs from the multiple feature vector sets; calculate the similarity between each feature vector in the candidate feature vector set and the target feature vector; and select the minor part category corresponding to the target candidate feature vector with the highest similarity as the part type.

[0158] Optionally, if there are multiple target candidate feature vectors with the highest similarity, the intelligent sorting decision module 25 is further configured to calculate the comprehensive weights of the target candidate feature vectors based on the historical matching accuracies of the target candidate feature vectors and the ambient light data, where the historical matching accuracy is the statistical result of the correct and incorrect sorting of the standard parts under different ambient light data; and select the minor part category corresponding to the target candidate feature vector with the highest comprehensive weight as the part type.

[0159] Optionally, the intelligent sorting decision module 25 is further configured to generate a grasping strategy according to the geometric features of the part, where the grasping strategy includes a grasping position and a grasping angle; plan the motion path of the robotic arm based on the target position information corresponding to the part type and the grasping strategy; and control the robotic arm to perform part sorting according to the grasping strategy and the motion path.

[0160] It should be noted that when the device provided in the above embodiments realizes its functions, only the division of the above functional modules is used for illustration. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept. For the specific implementation process, please refer to the method embodiments and will not be elaborated here.

[0161] This embodiment also discloses an electronic device 900. Referring to Figure 10 , the electronic device may include: at least one processor 901, at least one communication bus 902, a user interface 903, a network interface 904, and at least one memory 905.

[0162] Among them, the communication bus 902 is used to realize the connection and communication between these components.

[0163] Among them, the user interface 903 may include a display screen (Display) and a camera (Camera). Optionally, the user interface may further include a standard wired interface and a wireless interface.

[0164] Among them, the network interface 904 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0165] Among them, the processor 901 may include one or more processing cores. The processor uses various interfaces and lines to connect various parts within the entire server, and by running or executing instructions, programs, code sets, or instruction sets stored in the memory, as well as calling data stored in the memory, it executes various functions of the server and processes data. Optionally, the processor may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor may integrate a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. in a combination of one or more. Among them, the CPU mainly processes the operating system, user interface, and application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above modem may not be integrated into the processor and may be implemented separately through a single chip.

[0166] Among them, the memory 905 may include a Random Access Memory (RAM), or may also include a Read-Only Memory. Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above method embodiments, etc.; the data storage area may store the data involved in the above method embodiments. Optionally, the memory may also be at least one storage device located far from the aforementioned processor. As shown in the figure, the memory as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program for a part sorting method based on AR glasses.

[0167] In Figure 10 In the electronic device shown, the user interface is mainly used to provide an input interface for the user and obtain the data input by the user; while the processor can be used to call the application program for a part sorting method based on AR glasses stored in the memory. When executed by one or more processors, the electronic device executes the methods in one or more of the above embodiments.

[0168] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0169] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0170] In several embodiments provided by this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some service interfaces. The indirect coupling or communication connection of the device or unit can be in electrical or other forms.

[0171] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, each functional unit in various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0173] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of this application. The aforementioned memory includes: various media such as USB flash drives, mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0174] The above are only exemplary embodiments of the present disclosure and should not be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure still fall within the scope covered by the present disclosure. Those skilled in the art will readily think of other implementation schemes of the present disclosure after considering the disclosure of the specification. This application aims to cover any variations, uses, or adaptive changes of the present disclosure. These variations, uses, or adaptive changes follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the technical field not recorded in the present disclosure. The specification and embodiments are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A part sorting method based on AR glasses, characterized in that, The method includes: Collecting ambient light data; Based on the ambient light data, adjusting the parameters of the data acquisition module, the data acquisition module includes a camera, a depth sensor, and a spectral sensor, and the parameters include the exposure time of the camera, the scanning accuracy of the depth sensor, and the sampling interval of the spectral sensor; Obtaining multi-modal data of the part through the data acquisition module, the multi-modal data includes image data of the part, point cloud data of the part, and spectral data of the part, and the part is a precision electronic component; Generating a target feature vector based on the multi-modal data and the weights corresponding to each modal data; Based on the feature vectors of different standard parts in the standard part feature library and the target feature vector, obtaining the part type, and controlling the robotic arm to complete part sorting according to the part type.

2. The method according to claim 1, characterized in that The generating a target feature vector based on the multi-modal data and the weights corresponding to each modal data specifically includes: Based on the image data of the part, extracting two-dimensional feature information of the part, the two-dimensional feature information includes edge contour, texture feature, and color distribution; Based on the point cloud data of the part, extracting three-dimensional feature information of the part, the three-dimensional feature information includes surface curvature, size ratio, and three-dimensional shape; Based on the spectral data of the part, extracting material feature information of the part, the material feature information includes reflectivity, absorptivity, and scattering characteristics; Evaluating the quality of each modal data, and adjusting the weights of each modal feature information according to each quality, and the modal feature information includes two-dimensional feature information, three-dimensional feature information, and material feature information; Generating a target feature vector based on the modal feature information and the weights of the modal feature information.

3. The method according to claim 2, wherein The extracting three-dimensional feature information of the part based on the point cloud data of the part specifically includes: Preprocessing the point cloud data to obtain preprocessed point cloud data; Constructing a three-dimensional model based on the preprocessed point cloud data; Obtaining the three-dimensional feature information based on the three-dimensional model.

4. The method according to claim 2, wherein The evaluating the quality of each modal data specifically includes: Calculating the signal-to-noise ratio of each modal data; Evaluating the discrimination of each modal data based on a preset part sample library, and the discrimination is the difference degree between each modal data of the part and the corresponding standard data in the part sample library; Detecting the integrity of each modal data, and the integrity is the ratio of the quantity of the actually obtained each modal data to the total number of items of each modal standard data; Determining the data quality of each modal data based on the signal-to-noise ratio, discrimination, and integrity.

5. The method according to claim 1, wherein The obtaining the part type based on the feature vectors of different standard parts in the standard part feature library and the target feature vector specifically includes: Based on the feature vectors of different standard parts in the standard part feature library, classifying the standard parts to obtain a plurality of feature vector sets, each feature vector set corresponds to a major part category, and each major part category includes at least one minor part category, and the minor part category is the part type; Determining the candidate feature vector set to which the target feature vector belongs from the plurality of feature vector sets; Calculate the similarity between each feature vector in the set of candidate feature vectors and the target feature vector; Select the part subclass corresponding to the target candidate feature vector with the highest similarity as the part type.

6. The method according to claim 5, characterized in that, The selection of the part subclass corresponding to the candidate feature vector with the highest similarity as the part type specifically includes: If there are multiple target candidate feature vectors with the highest similarity, calculate the comprehensive weights of the target candidate feature vectors based on the historical matching accuracies of the target candidate feature vectors and the ambient light data, where the historical matching accuracy is the statistical result of the correct and incorrect sorting of the standard parts under different ambient light data; Select the part subclass corresponding to the target candidate feature vector with the highest comprehensive weight as the part type.

7. The method according to claim 1, characterized in that, The control of the robotic arm to complete part sorting according to the part type specifically includes: Generate a grasping strategy based on the geometric features of the part, where the grasping strategy includes the grasping position and the grasping angle; Plan the motion path of the robotic arm based on the target position information corresponding to the part type and the grasping strategy; Control the robotic arm to perform part sorting according to the grasping strategy and the motion path.

8. A part sorting system based on AR glasses, characterized in that, Specifically includes: An ambient light perception module for collecting ambient light data; An adaptive parameter regulation module for adjusting the parameters of the data acquisition module based on the ambient light data, where the data acquisition module includes a camera, a depth sensor, and a spectral sensor, and the parameters include the exposure time of the camera, the scanning accuracy of the depth sensor, and the sampling interval of the spectral sensor; A multi-modal synchronous acquisition module for acquiring multi-modal data of the part through the data acquisition module, where the multi-modal data includes the image data of the part, the point cloud data of the part, and the spectral data of the part, and the part is a precision electronic component; A feature fusion engine module for generating a target feature vector based on the multi-modal data and the weights corresponding to each modal data; An intelligent sorting decision module for obtaining the part type based on the feature vectors of different standard parts in the standard part feature library and the target feature vector, and controlling the robotic arm to complete part sorting according to the part type.

9. An electronic device, characterized in that, It includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions. Both the user interface and the network interface are used to communicate with other devices. The processor is used to execute the instructions stored in the memory so that the electronic device executes the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1-7 is executed.

Images