Ground material identification method, system, and self-moving device

Abstract

The application relates to a ground material identification method, system and self-moving device. The method comprises the following steps: collecting an image containing a ground surface by a near-infrared imaging method; pre-processing the collected image to obtain a region of interest, wherein the region of interest is a target region determined from the image; performing feature extraction on the region of interest; and performing classification identification of the ground material according to the feature extraction result. The scheme of the application can realize accurate identification of the ground material on a low-computing-power platform.

Description

Technical Field

[0001] This invention relates to the field of sweeping machine technology, and in particular to a method, system and self-moving device for identifying floor materials. Background Technology

[0002] With the advent of the information age and the development of IoT and AI technologies, a large number of efficient and convenient smart appliances have entered people's lives, greatly facilitating daily life and freeing ordinary people from heavy and repetitive housework. Among them, robotic vacuum cleaners are becoming increasingly popular with users due to their increasingly intelligent features and efficient cleaning capabilities.

[0003] In the use of self-moving devices, such as robotic vacuum cleaners, applying different operating logic based on different floor materials helps to better complete the cleaning task. For example, applying different amounts of water depending on the floor material. Therefore, to achieve this goal, it is first necessary to accurately identify the floor material of the current cleaning area.

[0004] Current common ground material identification methods are based on multimodal signal fusion, which combines several different signals (such as acoustic signals, visual images, etc.) and uses machine learning algorithms to identify materials. However, this method is complex due to the complexity of data acquisition and annotation, the difficulty of algorithm design, and the high computational requirements. Summary of the Invention

[0005] In view of this, the present disclosure aims to provide a ground material identification method, system and self-moving device to achieve accurate identification of ground materials even on low computing power platforms.

[0006] According to this disclosure, a method for identifying ground material is first provided, the method comprising:

[0007] Images containing the ground were acquired using near-infrared imaging.

[0008] The acquired image is preprocessed to obtain a region of interest, which is a target region determined from the image;

[0009] Feature extraction is performed on the region of interest;

[0010] The ground material is classified and identified based on the feature extraction results.

[0011] According to one embodiment of the present invention, the classification and identification includes: employing a two-stage material identification method based on SVM or employing a multi-classification end-to-end identification method. The two-stage material identification method based on SVM performs step-by-step identification by setting classifiers in two stages respectively, while the multi-classification end-to-end identification method performs identification by setting a classifier in one stage.

[0012] According to one embodiment of the present invention, when using a two-stage material identification method based on SVM, in the first stage, the target data is distinguished based on the feature extraction results of the target data to determine whether the target data is of a type whose attributes or categories are different from the remaining data types in the database. If so, the target data type is identified; otherwise, the second stage is executed. In the second stage, the target data is further distinguished based on the feature extraction results of the target data to determine which of the other types the target data belongs to.

[0013] According to one embodiment of the present invention, the preprocessing includes: obtaining the region of interest using statistical methods.

[0014] According to one embodiment of the present invention, obtaining the region of interest using statistical methods includes:

[0015] The dataset containing ground features was obtained by filtering the collected semantic segmentation labeled data.

[0016] Statistically analyze the longitudinal boundary curves of the ground region in the dataset, and calculate the mean and variance of the dataset.

[0017] Based on the mean and variance, the boundary of the dataset of the selected ground areas is approximately fitted to a Gaussian distribution to identify the area with the most concentrated energy.

[0018] Based on the obtained mean, variance, and Gaussian distribution, statistical analysis is performed through normal distribution modeling to obtain an approximate outer boundary. The width and height of this outer boundary are the width and height of the bounding box of the region of interest.

[0019] The target region is determined by using the width and height values ​​of the image acquired by the near-infrared imaging method. The target region is a rectangular frame with the bottom center of the acquired image as the center of the bottom edge. The width and height values ​​are the width and height values ​​of the rectangular frame.

[0020] According to one embodiment of the present invention, the feature extraction includes extracting frequency domain features using Fast Fourier Transform, extracting color features using HSV color space domain, extracting texture features using local binary mode, or obtaining feature values ​​using a multi-feature fusion method.

[0021] According to one embodiment of the present invention, multi-feature fusion is performed by controlling the proportion of the modulus of multiple features of the separately extracted region of interest to the modulus of the total features, or by controlling the proportion of the values ​​of the multiple features to the total feature values.

[0022] The present invention also proposes a ground material identification system, the system comprising:

[0023] The image acquisition module is configured to acquire images containing the ground using near-infrared imaging.

[0024] The preprocessing module is configured to preprocess the acquired image to determine the target region from the image and obtain the region that needs further processing, i.e., the region of interest.

[0025] The feature extraction module is configured to extract features from the region of interest;

[0026] The identification module is configured to classify and identify ground materials based on the feature extraction results.

[0027] According to one embodiment of the present invention, the feature extraction module performs feature extraction by: extracting frequency domain features using fast Fourier transform, or extracting color features using the HSV color space domain, or extracting texture features using local binary mode, or obtaining feature values ​​by multi-feature fusion.

[0028] The identification module performs classification and identification by adopting a two-stage material identification method based on SVM. In the first stage, the target data is distinguished based on the feature extraction results of the target data to determine whether the target data is a type that can be clearly distinguished from other types of data. If so, the target data type is identified; otherwise, the second stage is executed. In the second stage, the target data is further distinguished based on the feature extraction results of the target data to determine which of the other types the target data belongs to.

[0029] The present invention also proposes a self-moving device, which includes a processor and a memory, wherein the memory stores program instructions, and the processor executes the program instructions to implement the steps in any of the methods described above.

[0030] This invention proposes a near-infrared imaging method for data identification, enabling the identification of ground material types on low-computing-power platforms. Compared to acoustic signal methods, visual solutions obviously provide richer semantic information and have better robustness. Furthermore, the various ground material identification methods combined with machine learning algorithms designed in this invention can be tailored to the specific characteristics of the identification category and, based on the computing platform, can employ a variety of different feature recognition and classification recognition methods. This is more efficient than existing ground material identification methods in the robotic vacuum cleaner field and can achieve accurate material identification even on low-computing-power platforms (such as the MR813 chip).

[0031] To further understand the features and technical content of this invention, please refer to the following detailed description and accompanying drawings. However, these descriptions and drawings are only for illustrating the invention and are not intended to limit the scope of protection of the invention in any way. Attached Figure Description

[0032] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings. These drawings, which form part of this disclosure, are used to provide a further understanding of the disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain the disclosure and do not constitute an undue limitation of the disclosure. In the drawings:

[0033] Figure 1 This diagram illustrates a flow chart of a ground material identification method according to an example embodiment of this application.

[0034] Figure 2 This diagram illustrates the Gaussian distribution of the ground region boundary according to an example embodiment of this application.

[0035] Figure 3 This diagram illustrates the use of statistical methods to extract regions of interest according to an example embodiment of this application.

[0036] Figure 4 A schematic diagram of a two-stage reasoning framework according to an example embodiment of this application is shown;

[0037] Figure 5 A schematic diagram of an end-to-end multi-feature fusion framework according to an example embodiment of this application is shown. Detailed Implementation

[0038] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0039] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of these specific details, or other methods, components, materials, devices, etc. In these cases, well-known structures, methods, devices, implementations, materials, or operations will not be shown or described in detail.

[0040] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0041] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0042] This invention proposes a vision-based method for identifying floor materials. The vision sensor utilizes a near-infrared imaging device (which is significantly cheaper than a color camera), enabling the identification of floor material types (including but not limited to carpets, wooden floors, and tiles) on low-computing-power platforms. Compared to acoustic signal methods, the vision approach obviously provides richer semantic information and exhibits better robustness. Furthermore, the floor material identification method incorporating machine learning algorithms designed in this invention is more efficient than existing floor material identification methods in the robotic vacuum cleaner field, achieving accurate material identification even on low-computing-power platforms (such as the MR813 chip).

[0043] The embodiments of the technical solution of the present invention will be described in detail below.

[0044] First, this invention presents a highly efficient and accurate method for identifying ground materials based on near-infrared imaging. For example... Figure 1 As shown, the method mainly includes several parts: step S100 data acquisition, step S110 data preprocessing (extracting regions of interest from the image), step S120 feature extraction, and step S130 classification and recognition.

[0045] Step S100 data acquisition involves acquiring images containing the ground using near-infrared imaging.

[0046] After data collection, to identify the ground material, we first need to know which part of an image is the ground. In visual processing, this means finding the region of interest, which is step S110 data preprocessing.

[0047] To obtain the region of interest, according to one embodiment of the present invention, the present invention can use statistical methods to calculate the region in a dataset most likely to contain the ground, thereby obtaining the region of interest.

[0048] The statistical method for extracting regions of interest (ROIs) is as follows: A ROI extraction method is designed based on prior statistical knowledge of the scene. Specifically, by analyzing data collected by mobile devices such as robotic vacuum cleaners, it is found that the ground area is mainly concentrated in the lower half of the image. Through statistical analysis, statistically significant boundary information is obtained, i.e., a "bounding box" of a certain size is determined based on the statistical information. This bounding box is used to confirm the ROI, which is then fed into the algorithm for processing and recognition. Using statistical learning methods to analyze visual data of ground materials enables reliable models built based on massive data analysis and statistics to be deployed in production environments.

[0049] A significant advantage of statistical learning methods is their simplicity and speed in the inference phase (i.e., the production phase). Utilizing statistical prior knowledge derived from the collected data, an approximate region of interest (ROI) can be obtained by setting a few simple parameters (practice has shown that such priors have good robustness and practicality). A potential drawback of this method is that the extracted ROI is not a completely accurate ground-level ROI, but rather an approximate ROI containing background information (defined here as other areas not belonging to the ground). However, on the other hand, since the training data is ROI data containing background information, to a certain extent, the model trained in this way will exhibit the same reliability and robustness when facing ROI data with similar distributions during inference.

[0050] When using statistical methods to extract regions of interest, it is required to perform statistics on a dataset that is universal and extensive. Therefore, in order to ensure that the statistically obtained data is sufficiently representative, data can be collected and organized from several dimensions: region (north, south, east, west), season, house type, etc., and uniform sampling should be performed according to different types of data to ensure that the complete dataset contains data of each type that are approximately the same in magnitude.

[0051] According to one embodiment of the present invention, the statistical method for extracting regions of interest may further include the following steps:

[0052] The dataset containing ground features was obtained by filtering the collected semantic segmentation labeled data.

[0053] Statistically analyze the longitudinal boundary curves of the ground region in the dataset, and calculate the mean and variance of the dataset.

[0054] Based on the mean and variance, the boundaries of the selected ground regions are approximately fitted to a Gaussian distribution to identify the regions with the highest energy concentration (e.g., Figure 2 The boundary region of the mid-coordinate is [-2σ: 2σ], which concentrates approximately 95% of the energy.

[0055] Based on the obtained mean, variance, and Gaussian distribution, statistical analysis is performed using normal distribution modeling to obtain an approximate outer boundary. The width and height of this outer boundary are the width and height of the bounding box of the region of interest. An approximate outer boundary can be obtained using formula (1), and the size of this region is the size of the region of interest, such as... Figure 3 As shown;

[0056] The target region is determined by using the width and height values ​​of the image acquired by the near-infrared imaging method. The target region is a rectangular frame with the bottom center of the acquired image as the center of the bottom edge. The width and height values ​​are the width and height values ​​of the rectangular frame.

[0057]

[0058] In the above formula, x is the target object of the statistical sample, σ is the variance of the target object, μ is the expected value of the statistical sample of the target object, and f(x) is the probability density function of the normal distribution (Gaussian distribution function).

[0059] In other words, based on a dataset containing ground region masks, statistical analysis is performed on this dataset to derive a relatively scientifically accurate boundary for the ground region. The specific steps are as follows:

[0060] Statistically analyze the width and height distribution of the ground area in this dataset;

[0061] Calculate the mean and variance of the width and height of this dataset;

[0062] Calculate statistical width and height: For example, the upper limit of statistical height is h. s =h μ +2h ρ The statistically wide boundary is: the left boundary is w. l =w μ -2w ρ The right boundary is w r =w μ +2w ρ Among them, h ρand w ρ The magnitude of h is determined by the Gaussian distribution curve, and a reasonable value is determined based on the energy accumulation law. s h represents the height of the region of interest μ and h ρ w represents the high mean and variance obtained from statistical data. l w represents the left boundary of the region of interest. r w represents the right boundary of the region of interest. μ and w ρ This represents the mean and variance of the width, statistically derived from the collected sample data.

[0063] The following section explains the combination of material identification and feature extraction.

[0064] According to one embodiment of the present invention, the feature extraction in step S120 can employ Fast Fourier Transform (FFT) to extract frequency domain features, or hsv(Hue, Saturation, Value) color space domain to extract image color features, or Local Binary Patterns (lbp) algorithm, etc. These algorithms can all be implemented using existing technologies and will not be detailed here.

[0065] According to one embodiment of the present invention, the present invention can combine multi-feature fusion algorithms and machine learning to identify ground materials.

[0066] According to one embodiment of the present invention, during the classification and identification step S130, the present invention may employ a two-stage material identification method based on SVM (Support Vector Machine).

[0067] The identification of floor materials mainly includes wooden floors, tiles, and carpets. To achieve efficient and reliable floor material identification, this application has confirmed through preliminary scheme verification that SVM is used as the learning model. SVM is an efficient binary classification machine learning algorithm. Furthermore, since some floor materials are significantly different from others—for example, carpets visually differ significantly from both wooden floors and tiles—a two-stage machine learning algorithm is designed.

[0068] A two-stage learning algorithm based on SVM logically designs a method for material recognition. According to one embodiment of the present invention, such as... Figure 4As shown, the first stage trains an SVM-based binary classifier, which is mainly used to distinguish whether the target data is significantly different from other types of data, such as whether the ground is carpet. The second stage is based on the reasoning results of the first stage. That is, if the data is not significantly different from other types, such as carpet, then it is sent to the second classifier for identification to further distinguish which type of the remaining types the target is, such as flooring or tile.

[0069] Clearly, the two stages described above employ different classifiers. Specifically, classifiers for each stage are trained separately during the training phase; for example, one classifier distinguishes between carpet and non-carpet, while the other distinguishes between tile and flooring. One advantage of this method is that it better aligns with the characteristics of the native SVM binary classifier, dividing the problem domain into two smaller sub-problem sets, thus making it easier to achieve good recognition results.

[0070] In the above recognition scheme, the feature extraction algorithms in the two stages can be flexibly configured (ensuring consistency between the training and inference stages). After verification, the Fast Fourier Transform Frequency Domain Feature Extraction (FFT), HSV Color Space Domain Feature Extraction, and LBP Local Binary Pattern Feature Extraction algorithms all have good performance.

[0071] Fast Fourier Transform (FFT) extracts features by extracting frequency domain features; HSV feature extraction extracts color features from images; and Local Binary Pattern Extraction (LBPI) is an operator used to describe local texture features of an image, possessing significant advantages such as rotation invariance and grayscale invariance. The specific algorithm can be flexibly selected based on differences in production equipment and computing resources (FFT consumes more CPU resources compared to other feature extraction algorithms).

[0072] According to one embodiment of the present invention, the present invention can also employ an SVM-based end-to-end multi-feature fusion algorithm for feature extraction and classification recognition, so as to combine the feature extraction in step S120 and the classification recognition in step S130.

[0073] The SVM-based end-to-end multi-feature fusion algorithm mainly includes two aspects. One aspect refers to the fact that the entire reasoning and recognition process can be completed in one step. During recognition, a multi-classifier is used to distinguish different materials. The algorithm framework is as follows: Figure 5As shown. On the other hand, it refers to the use of multi-feature fusion in the feature extraction stage. The advantage of doing so is that a single feature extractor can only extract one aspect of the input data, such as texture features or color features. It is an obvious fact that many visual recognition scenarios cannot be recognized by just one feature. They often need to comprehensively consider features such as contour, color and texture. Based on this idea, this application designs a method that integrates multiple feature extractions, such as a multi-feature fusion method of color features and texture features. The feature fusion method is shown in Equation (2):

[0074]

[0075] Where F f F represents the fusion feature. t This represents texture features, F c This represents color features, with α and β being weighting parameters.

[0076] To obtain more representative fused features, feature fusion can control the proportion of each feature in the fused feature from several aspects:

[0077] On the one hand, this is achieved by controlling the ratio of the magnitude of multiple feature vectors to the magnitude of the fused feature vector, that is, the proportion of the dimensions of the fused feature vector occupied by multiple vectors. For example, let the size of the fused feature vector be L. f The vector size of the texture feature is L t The vector size of the color feature is L c The relationship between the three is shown in formula (3):

[0078] L f =λL t +γL c (3)

[0079] Where λ represents the length of the texture feature vector and γ represents the length of the color feature vector.

[0080] λ and γ are used to control the size of the fused feature vectors. The representativeness of the model can be adjusted by controlling the length of each feature vector.

[0081] On the other hand, it controls the proportion of each feature vector value to the total feature vector value, as shown in formula (2).

[0082] These two control methods can effectively adjust the weight ratio of different features in the fusion vector, thereby obtaining better feature extraction values.

[0083] Of course, the above control algorithm only gives an example of the fusion of texture features and color features. It can also be a combination of other feature extraction algorithms, and it is not limited to the fusion of two features. It can be the fusion of three or more features. For example, texture features and color features can be combined with features such as frequency domain features for feature fusion.

[0084] The material identification method of the present invention may further include voting identification.

[0085] In the application scenario of robotic vacuum cleaners, material recognition is actually performed as a local whole. In short, the goal is to infer the material type of each room based on the already partitioned room and the location information (pos information).

[0086] According to one embodiment of the present invention, material identification can be logically divided into two main stages. The first stage is high-frequency real-time inference, where the self-moving device, such as a robot vacuum cleaner, moves around the room and uses a material identification algorithm (i.e., the aforementioned SVM-based multi-feature fusion algorithm, etc.) at a certain frequency to infer the floor material type and saves the corresponding timestamp and identification results. The second stage is the voting identification stage. The second stage is implemented after the self-moving device, such as the robot vacuum cleaner, has finished cleaning. At this time, we have the floor material identification result values ​​of the entire cleaning process of the self-moving device and the corresponding room partitions. The material identification detection points in a single room are treated as a set for voting, and the floor material type of each room is finally confirmed based on the voting results. That is, the identification results of multiple identifications in a single room are statistically analyzed, and the material type with the most votes is determined as the identification result of that room. For example, as the self-moving device moves, suppose a room is inferred n times in total, where the n1th result is material A, the n2th result is material B, and n1 + n2 = n. Then, the material with the larger value between n1 and n2 is taken as the identification result, which is the voting result.

[0087] Related processing logic: ① Filtering logic. This logic is mainly used to filter invalid data, based on two conditions. The first is the orientation of the self-moving device, i.e., filtering is based on the orientation of the self-moving device at each material identification monitoring point at the corresponding time. If the orientation is facing the wall, the first condition is met. The second condition is boundary logic, i.e., when the distance between the position of the self-moving device at the corresponding material identification detection point and the boundary is less than a set value Db, the second condition is met. When a material identification detection point meets both of these conditions, the data is considered invalid and will not participate in the voting (this situation is prone to misidentification).

[0088] ② Invalid Voting Judgment Logic. An invalid vote refers to a situation where the reliability of the voting result is considered poor, in which case the material type of that area or building is forcibly identified as the default material. When the number of valid material identification detection points in a single room or area is less than a certain threshold, it is considered that the room's observation points are insufficient to produce a reliable identification result, and the room's floor material is set as the default material category.

[0089] ③ Not involved in scene recognition. Based on prior statistical knowledge, it is known that some rooms are defaulted to a certain material. This mainly refers to the bathroom scene. When the mobile device cleans the bathroom, the material recognition function will not be activated and the room will be set to the tile type by default after cleaning is completed.

[0090] The following examples illustrate embodiments of the present invention.

[0091] According to one embodiment of the present invention, for example, the self-moving device acquires an image containing the ground during the cleaning process using near-infrared imaging, wherein the ground in the image is wooden floor data, and the data is first preprocessed.

[0092] When using statistical prior methods to extract regions of interest (ROIs), it is necessary to statistically analyze the prior information of ROIs for different ground materials on a dataset with labeled ground regions. This involves determining the boundaries of the ground regions based on the statistical results of large datasets (the statistical target here is the length and width of the ground region relative to the bottom center point of the image), thus obtaining the ROI of the ground region (e.g., ...). Figure 3 As shown, the bright-colored area is the region of interest for the ground material.

[0093] Next, feature extraction and classification are performed. The region of interest is fed into the model trained by machine learning, and the multi-feature fusion algorithm described above is used for feature extraction and classification. This completes the feature extraction and identification of the ground material, determining the type of ground material in the area where the current data is located.

[0094] Because near-infrared imaging equipment has a significantly lower imaging quality compared to color imaging equipment, it contains less feature information. Therefore, this implementation prioritizes extracting texture and frequency domain information from the region of interest (ROI) during the feature extraction stage, rather than utilizing the color information lacking in near-infrared imaging. This approach better extracts feature information from the ground region. Specifically, a Local Binary Pattern (LBP) algorithm was developed for texture information extraction, and a Fast Fourier Transform (FFT) algorithm was used for frequency domain feature extraction to extract the frequency domain features of the ROI. After feature extraction, a feature fusion approach was employed, combining both texture and frequency domain features to improve the algorithm's robustness and achieve more reliable recognition results.

[0095] The recognition results of the above implementation methods using different feature extraction methods are shown in the table below.

[0096] Table 1

[0097]

[0098] The above method can be deployed and run on low-computing-power platforms such as the MR813 chip computing platform, and has the ability to deploy and accurately identify on low-computing-power platforms.

[0099] This application also discloses a ground material identification system, the system comprising:

[0100] The image acquisition module is configured to acquire images containing the ground using near-infrared imaging.

[0101] A preprocessing module is configured to preprocess the acquired image to obtain the region of interest;

[0102] The feature extraction module is configured to extract features from the region of interest;

[0103] The identification module is configured to classify and identify ground materials based on the feature extraction results.

[0104] According to one embodiment of the present invention, the feature extraction module performs feature extraction by: extracting frequency domain features using fast Fourier transform, or extracting color features using the HSV color space domain, or extracting texture features using local binary mode, or obtaining feature values ​​by multi-feature fusion.

[0105] According to one embodiment of the present invention, the identification module performs classification identification by: adopting a two-stage material identification method based on SVM, wherein, in the first stage, it is distinguished whether the target data is a type that clearly distinguishes it from other types of data; if so, the target data type is identified; if not, the second stage is executed; in the second stage, it is further distinguished which of the other types the target data belongs to.

[0106] Each module in the ground material identification system of the present invention can execute any corresponding function of the above method.

[0107] The present invention also proposes a self-moving device, which includes a processor and a memory, wherein the memory stores program instructions, and the processor executes the program instructions to implement any step of the method described above.

[0108] The self-moving device described in this invention is not limited to a sweeping robot, but can also be an automatic device such as a floor scrubber or floor cleaner.

[0109] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0110] Compared with existing visual analysis methods, the technical solution provided in this application is a complete set of efficient technologies based on machine learning and visual analysis, with better robustness and extremely high practicality. In applications using self-mobile devices, it can achieve accurate identification of floor material categories (including but not limited to carpet, wooden flooring, and tile). Furthermore, this method can be extended to other visual recognition and classification scenarios.

[0111] Finally, it should be noted that the above descriptions are merely exemplary embodiments of this disclosure and are not intended to limit this disclosure. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A ground material identification method characterized by comprising: The method is applied to a self-moving device, and the method comprises: Collecting an image containing a ground surface by using a near-infrared imaging method; Preprocessing the collected image to obtain a region of interest; the region of interest is a target region determined from the image; the preprocessing comprises obtaining the region of interest by using a statistical method, which comprises: screening a data set containing a ground surface according to labeled data of semantic segmentation that has been collected; statistically determining a longitudinal boundary curve of a ground surface region of the data set, and calculating a mean value and a variance on the data set; approximating the boundary of the screened data set of the ground surface region to a Gaussian distribution according to the mean value and the variance, and confirming a region with the most concentrated energy; based on the obtained mean value and variance and Gaussian distribution, performing statistical analysis by normal distribution modeling to obtain an approximate outer boundary, and a width value and a height value of the outer boundary are width and height values of a bounding box of the region of interest; using the width value and the height value to determine a target region of the image collected by the near-infrared imaging method, the target region being a rectangular box with a bottom center of the collected image as a bottom center, and the width value and the height value being width and height values of the rectangular box; Performing feature extraction on the region of interest; Performing classification and identification of a ground surface material according to the feature extraction result; The method further comprises voting identification, which is performed after the self-moving device completes cleaning, at which time we have ground surface material identification result values and corresponding room partitions of the entire cleaning process of the self-moving device, and the material identification detection points in a single room are taken as a set for voting, and finally the ground surface material types of each room are confirmed according to the voting result; relevant processing logic comprises: Filtering logic: this logic is used to filter invalid data, and is based on two conditions, the first being the orientation of the self-moving device, that is, filtering according to the orientation of the self-moving device at the corresponding time of each material identification monitoring point, and the second condition being a marginal logic, that is, when the distance between the position of the self-moving device at the corresponding time of the material identification detection point and the boundary is less than a set value Db, the second condition is satisfied; when the material identification detection point satisfies both conditions, the data is considered to be an invalid data point and does not participate in voting; Invalid voting determination logic: invalid voting refers to a situation where the reliability of the voting result is poor, and the material type of the region or house is forcibly determined as a default material; Non-identification scene: according to prior statistical knowledge, it is known that some rooms are by default a certain material.

2. The ground material identification method according to claim 1, wherein The classification and identification comprises: using a two-stage material identification method based on svm or using a multi-class end-to-end identification method, the two-stage material identification method based on svm identifies in steps by setting classifiers in two stages, and the multi-class end-to-end identification method identifies by setting a classifier in one stage.

3. The ground material identification method according to claim 2, wherein In the two-stage material recognition method based on SVM, in the first stage, whether the target data is of a type different from the remaining data types in the database is determined according to the feature extraction result of the target data, if yes, the type of the target data is recognized, and if not, the second stage is performed; in the second stage, which type the target data belongs to among the other types is further determined according to the feature extraction result of the target data.

4. The ground material identification method according to claim 1 or 2 or 3, characterized by, The feature extraction includes extracting frequency domain features by using fast Fourier transform, extracting color features by using an HSV color space domain, extracting texture features by using a local binary pattern, or obtaining feature values by using a multi-feature fusion manner.

5. The ground material identification method according to claim 4, wherein The multi-feature fusion manner includes: controlling the proportion of the number of features of the region of interest extracted alone in the total number of features, or controlling the proportion of the values of the plurality of features in the total feature value.

6. A ground material identification system characterized by, The system is applied to a self-moving device, and the system comprises: An image acquisition module configured to acquire an image containing a floor by using a near-infrared imaging method; A preprocessing module configured to preprocess the acquired image to determine a target region from the image to obtain a region of interest; the preprocessing includes obtaining the region of interest by using a statistical method, which includes: screening a data set containing a floor according to labeled data of semantic segmentation that has been acquired; statistically determining a longitudinal boundary curve of a floor region of the data set, and calculating a mean value and a variance on the data set; approximating the boundary of the screened data set of the floor region to a Gaussian distribution according to the mean value and the variance to confirm a region with the most concentrated energy; based on the obtained mean value and variance and Gaussian distribution, statistically analyzing by using normal distribution modeling to obtain an approximate outer boundary, and the width and height of the outer boundary are the width and height of a bounding box of the region of interest; using the width and height to determine a target region from the image acquired by using the near-infrared imaging method, the target region being a rectangular box with a bottom center of the acquired image as a bottom center, and the width and height being the width and height of the rectangular box; A feature extraction module configured to extract features from the region of interest; An identification module configured to classify and identify the floor material according to the feature extraction result; The system further comprises voting identification, which is performed after the self-moving device completes cleaning, at this time, the self-moving device has the floor material identification result values and corresponding room partitions of the entire cleaning process of the self-moving device, the material identification detection points in a single room are taken as a set for voting, and finally the floor material types of each room are determined according to the voting result; the related processing logic includes: Filtering logic: the logic is used to filter invalid data, based on two conditions, the first is the orientation of the mobile device, that is, according to the orientation of the mobile device at the moment corresponding to each material identification monitoring point, the second condition is the marginal logic, that is, when the position of the mobile device at the moment corresponding to the material identification monitoring point is less than the set value Db from the boundary, the second condition is met; When the material identification monitoring point meets the two conditions at the same time, it is considered that the data is an invalid data point and does not participate in voting; Invalid voting judgment logic: invalid voting refers to the situation that the voting result is considered to be less reliable, and then the material type of the region or house is forcibly identified as the default material; Non-participation identification scene: according to prior statistical knowledge, it is known that some rooms are defaulted to a certain material.

7. The ground material identification system of claim 6, wherein the feature extraction module performs feature extraction including: extracting frequency domain features using fast Fourier transform, or extracting color features using hsv color space domain, or extracting texture features using local binary pattern, or obtaining feature values using multi-feature fusion; The identification module performs classification and identification including: using a two-stage material identification method based on svm, wherein in the first stage, it is determined whether the target data is a certain type with different attributes or categories from the remaining data types in the database according to the feature extraction result of the target data, if yes, the target data type is identified; If not, the second stage is executed; In the second stage, it is further determined which type of the remaining data types the target data belongs to according to the feature extraction result of the target data. The self-moving device includes a processor and a memory, the memory stores program instructions, and the processor executes the program instructions to realize the steps in the method of any one of claims 1 to 5.

8. A self-moving device, characterized in that, ​

Images