Self-learning identification method and device for self-mobile equipment, self-mobile equipment, and medium

Abstract

The present application discloses a self-learning recognition method, apparatus, self-mobile device, and medium for a self-mobile device. The method receives an object repair instruction and a repaired image of an object to be identified, and extracts similar features from the repaired image of the object to be identified according to the object repair instruction to obtain a first feature; wherein the repaired image of the object to be identified includes the object to be identified and repaired; when the self-mobile device detects that the object to be identified and repaired exists in an initial image obtained by photographing the area to be cleaned, similar features are extracted from the first image of the object to be identified and repaired to obtain a second feature; wherein the initial image includes the first image; and when the similarity between the first feature and the second feature is greater than or equal to a preset similarity threshold, the first image is updated. The present application can avoid misidentification and missed identification, and improve the cleaning accuracy of the self-mobile device.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a self-learning recognition method, device, self-mobile device, and medium for a self-mobile device. Background Art

[0002] Currently, AI object recognition technology has been widely used in sweeping robots, such as identifying objects such as shoes, socks, wires, pet feces, and scales, thereby realizing different modes of obstacle avoidance functions or corresponding displays on the app based on the different objects identified.

[0003] However, in open scenes, there are various objects, so there is often a certain similarity between certain features of the identified objects and the features of the objects to be identified, which can easily lead to misidentification by the self-mobile device. Once misidentified, the misidentification icon will be displayed on the app. In serious cases, it will directly lead to missed sweeping and mopping of the room. It may also lead to missed identification. For example, if socks or wires are not identified, they will be directly rolled into the self-mobile device, affecting the movement of the self-mobile device, resulting in low cleaning accuracy of the self-mobile device due to the easy occurrence of misidentification and / or missed identification. Summary of the Invention

[0004] The main purpose of this application is to provide a self-learning identification method, device, equipment and medium for a self-moving device, aiming to solve the current technical problem of low cleaning accuracy of the self-moving device due to misidentification or missed identification.

[0005] To achieve the above objectives, an embodiment of the present application provides a self-learning identification method for a mobile device, the self-learning identification method for a mobile device comprising:

[0006] receiving an instruction for repairing an object to be identified and a repaired image of an object to be identified, and performing similarity feature extraction on the repaired image of the object to be identified according to the instruction for repairing an object to be identified to obtain a first feature; wherein the repaired image of the object to be identified includes the repaired object to be identified;

[0007] When the mobile device detects that the object to be identified and repaired exists in an initial image obtained by photographing the area to be cleaned, similarity feature extraction is performed on the first image of the object to be identified and repaired to obtain a second feature; wherein the initial image includes the first image;

[0008] When the similarity between the first feature and the second feature is greater than or equal to a preset similarity threshold, the first image is updated.

[0009] Preferably, the step of updating the first image includes:

[0010] The first image is replaced by an object image corresponding to the second feature.

[0011] Preferably, the step of updating the first image further includes:

[0012] The first image is displayed, and the object category associated with the second feature is determined as the category of the object to be identified and repaired.

[0013] Preferably, the step of receiving an instruction to repair an object to be identified and a repaired image of the object to be identified, and extracting similar features from the repaired image of the object to be identified according to the instruction to repair an object to be identified to obtain a first feature comprises:

[0014] receiving the identified object repair instruction and the repaired image of the object to be identified;

[0015] If the identified object repair instruction is a misidentification self-learning repair instruction, similar features are extracted from the repair image of the object to be identified according to the misidentification self-learning repair instruction to obtain the first feature, and the first feature is stored in a preset feature database.

[0016] Preferably, the step of receiving the recognition object repair instruction and the repaired image of the object to be recognized, and extracting similar features from the repaired image of the object to be recognized according to the recognition object repair instruction to obtain the first feature further comprises:

[0017] receiving the identified object repair instruction and the repaired image of the object to be identified;

[0018] If the identified object repair instruction is a missed object repair instruction, performing similarity feature extraction on the repaired image of the object to be identified according to the missed object repair instruction to obtain the first feature, and acquiring category information of the first feature;

[0019] The first feature and the category information of the first feature are associated and stored in a preset feature database.

[0020] Preferably, when the similarity between the first feature and the second feature is greater than or equal to a preset similarity threshold, before the step of updating the first image, the step includes:

[0021] The feature vectors of the preset dimensions in the second feature are respectively compared with the feature vectors of the preset dimensions of the first feature pre-stored in the preset feature database to determine the similarity between the first feature and the second feature.

[0022] Preferably, before the step of receiving the recognition object repair instruction and the repaired image of the object to be recognized, and extracting similar features from the repaired image of the object to be recognized according to the recognition object repair instruction to obtain the first feature, the method further includes:

[0023] Obtain object images and their annotation information as training samples;

[0024] Build the initial feature extraction network;

[0025] Training the initial feature extraction network using a triplet loss function according to the training samples to obtain a pre-trained feature extraction network;

[0026] Receive an instruction to repair an object to be identified and a repaired image of the object to be identified;

[0027] Based on the pre-trained feature extraction network, similar features are extracted from the repaired image of the object to be identified according to the identification object repair instruction to obtain the first feature.

[0028] To achieve the above-mentioned purpose, the present application further provides a cleaning device for a self-moving device, the cleaning device for the self-moving device comprising:

[0029] a first feature extraction module configured to receive an instruction for repairing an object to be identified and a repaired image of the object to be identified, and extract similar features from the repaired image of the object to be identified according to the instruction for repairing the object to be identified to obtain a first feature; wherein the repaired image of the object to be identified includes the repaired object to be identified;

[0030] a second feature extraction module configured to extract similar features from the first image of the object to be identified and repaired to obtain a second feature when the mobile device detects that the object to be identified and repaired exists in an initial image obtained by photographing the area to be cleaned; wherein the initial image includes the first image;

[0031] An updating module is configured to update the first image when the similarity between the first feature and the second feature is greater than or equal to a preset similarity threshold.

[0032] Furthermore, to achieve the above-mentioned purpose, the present application also provides a self-mobile device, which includes a memory, a processor, and a cleaning program of the self-mobile device stored in the memory and runnable on the processor. When the cleaning program of the self-mobile device is executed by the processor, the steps of the self-learning identification method of the self-mobile device are implemented.

[0033] Furthermore, to achieve the above-mentioned purpose, the present application also provides a medium, which is a computer-readable storage medium, and the computer-readable storage medium stores a cleaning program for the self-mobile device, and when the cleaning program for the self-mobile device is executed by the processor, the steps of the self-learning identification method of the self-mobile device are implemented.

[0034] The present invention provides a self-learning recognition method, apparatus, self-mobile device, and medium for a self-mobile device. The method receives an object repair instruction and a repaired image of an object to be identified, and extracts similar features from the repaired image of the object to be identified according to the object repair instruction to obtain a first feature; wherein the repaired image of the object to be identified includes the repaired object to be identified, so as to obtain the first feature and provide a basis for subsequent similarity comparison; when the self-mobile device detects that the repaired object to be identified exists in an initial image obtained by photographing the area to be cleaned, similar features are extracted from the first image of the repaired object to be identified to obtain a second feature; wherein the initial image includes the first image, so as to obtain the second feature of the initial image and provide a basis for subsequent similarity comparison; when the similarity between the first feature and the second feature is greater than or equal to a preset similarity threshold, the first image is updated. By updating the first image, misidentification and missed identification are avoided, thereby improving the cleaning accuracy of the self-mobile device. During the cleaning process, the self-mobile device of the present invention first accurately identifies each object in the area to be cleaned, and then cleans according to the correct image or category of the object, thereby avoiding misidentification and missed identification and improving the cleaning accuracy of the self-mobile device. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the hardware operating environment involved in this application from a mobile device;

[0036] Figure 2 This is a flowchart of the first embodiment of the self-learning recognition method for a mobile device of the present application;

[0037] Figure 3 A flowchart of a specific embodiment of the self-learning recognition method for a mobile device of the present application;

[0038] Figure 4 A flowchart of another specific embodiment of the self-learning recognition method for a mobile device of the present application;

[0039] Figure 5 This is a schematic diagram of the functional modules of a preferred embodiment of the cleaning device for mobile equipment of the present application.

[0040] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0041] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0042] like Figure 1 As shown, Figure 1A possible application scenario provided for an embodiment of the present disclosure includes a self-moving device, which can be a cleaning robot, such as a sweeping robot, a mopping robot, etc. In some embodiments, the cleaning robot can be an automatic cleaning device, which can be an automatic sweeping robot or an automatic mopping robot. In implementation, the cleaning robot can be provided with a navigation system, which can detect and determine the working area and the specific position of the cleaning robot in the working area by itself. The cleaning robot can be provided with various sensors, such as infrared, laser and other sensors, for real-time detection of the ground debris status in the working area. In other embodiments, the automatic cleaning device can be provided with a touch-sensitive display to receive operation instructions input by the user. The automatic cleaning device can also be provided with a wireless communication module such as a WIFI module and a Bluetooth module to connect to the smart terminal and receive operation instructions transmitted by the user using the smart terminal through the wireless communication module. The automatic cleaning device includes a machine body 100, a perception system, a control system, a drive system 106, a cleaning system, an energy system, a human-computer interaction system and a memory.

[0043] The perception system includes a position determination device 102 located above the machine body, a collision sensor, an anti-fall sensor and an ultrasonic sensor, an infrared sensor, an accelerometer, a gyroscope, an odometer and other sensing devices 104 located in the forward part of the machine body, which provide the control system with various position information and motion state information and other environmental information of the machine. The position determination device includes but is not limited to a camera and a laser ranging device (LDS, i.e., a laser radar). In an embodiment of the present application, the position determination device adopts a laser radar device, which can be a single-line laser radar or a multi-line laser radar. The control system includes a processor, a memory and a cleaning program for a self-moving device stored in the memory and runnable on the processor; the processor in the control system can be used to call the cleaning program for the self-moving device stored in the memory and perform the following operations:

[0044] receiving an instruction for repairing an object to be identified and a repaired image of an object to be identified, and performing similarity feature extraction on the repaired image of the object to be identified according to the instruction for repairing an object to be identified to obtain a first feature; wherein the repaired image of the object to be identified includes the repaired object to be identified;

[0045] When the mobile device detects that the object to be identified and repaired exists in an initial image obtained by photographing the area to be cleaned, similarity feature extraction is performed on the first image of the object to be identified and repaired to obtain a second feature; wherein the initial image includes the first image;

[0046] When the similarity between the first feature and the second feature is greater than or equal to a preset similarity threshold, the first image is updated.

[0047] Furthermore, the step of updating the first image includes:

[0048] The first image is replaced by an object image corresponding to the second feature.

[0049] Furthermore, the step of updating the first image further includes:

[0050] The first image is displayed, and the object category associated with the second feature is determined as the category of the object to be identified and repaired.

[0051] Furthermore, the step of receiving the recognition object repair instruction and the repaired image of the object to be recognized, and extracting similar features from the repaired image of the object to be recognized according to the recognition object repair instruction to obtain the first feature includes:

[0052] receiving the identified object repair instruction and the repaired image of the object to be identified;

[0053] If the identified object repair instruction is a misidentification self-learning repair instruction, similar features are extracted from the repair image of the object to be identified according to the misidentification self-learning repair instruction to obtain the first feature, and the first feature is stored in a preset feature database.

[0054] Furthermore, the step of receiving the recognition object repair instruction and the repaired image of the object to be recognized, and extracting similar features from the repaired image of the object to be recognized according to the recognition object repair instruction to obtain the first feature further includes:

[0055] receiving the identified object repair instruction and the repaired image of the object to be identified;

[0056] If the identified object repair instruction is a missed object repair instruction, performing similarity feature extraction on the repaired image of the object to be identified according to the missed object repair instruction to obtain the first feature, and acquiring category information of the first feature;

[0057] The first feature and the category information of the first feature are associated and stored in a preset feature database.

[0058] Furthermore, when the similarity between the first feature and the second feature is greater than or equal to a preset similarity threshold, before the step of updating the first image, the step includes:

[0059] The feature vectors of the preset dimensions in the second feature are respectively compared with the feature vectors of the preset dimensions of the first feature pre-stored in the preset feature database to determine the similarity between the first feature and the second feature.

[0060] Furthermore, before the step of receiving the recognition object repair instruction and the repaired image of the object to be recognized, and extracting similar features from the repaired image of the object to be recognized according to the recognition object repair instruction to obtain the first feature, the method further includes:

[0061] Obtain object images and their annotation information as training samples;

[0062] Build the initial feature extraction network;

[0063] Training the initial feature extraction network using a triplet loss function according to the training samples to obtain a pre-trained feature extraction network;

[0064] Receive an instruction to repair an object to be identified and a repaired image of the object to be identified;

[0065] Based on the pre-trained feature extraction network, similar features are extracted from the repaired image of the object to be identified according to the identification object repair instruction to obtain the first feature.

[0066] To better understand the above technical solutions, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0067] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0068] Reference Figure 2 , Figure 2 The following is a flow chart of a self-learning identification method for a mobile device provided in the first embodiment of the present application. In this embodiment, the self-learning identification method for a mobile device includes the following steps:

[0069] Step S10 : receiving an instruction for repairing an object to be identified and a repaired image of the object to be identified, and performing similarity feature extraction on the repaired image of the object to be identified according to the instruction for repairing the object to be identified to obtain a first feature.

[0070] In this embodiment, the self-learning recognition method of the self-mobile device is applied to the self-mobile device, wherein the self-mobile device in this embodiment can be the above-mentioned sweeping robot, mopping robot, etc. Since the self-mobile device has been connected to a smart terminal such as a smartphone and a tablet computer, and the smart terminal can be provided with an application for controlling the self-mobile device and querying information, the user can interact with the self-mobile device through the application to control the operation of the self-mobile device and query various types of information of the self-mobile device. In addition, a camera is provided at the front end of the self-mobile device for detecting obstacles in the area pointed by the camera for cleaning and obstacle avoidance, thereby ensuring the working efficiency of the self-mobile device; and a feature extraction network is configured inside the self-mobile device, which can specifically be a similarity feature extraction network, and its type can specifically be CNN (Convolutional Neural Networks), which is used to extract features from the input image and output each feature in the image.

[0071] Specifically, after power-on, the mobile device can scan the area through the front-end camera, and generate a corresponding map after the scan is completed. For example, the area included in the user's three bedrooms and one living room is scanned, and the user's floor plan is obtained after the scan is completed. For example, a floor plan of three bedrooms and one living room is generated and displayed on the display interface of the application, and specific area size information is displayed, such as showing that the total area of ​​the three bedrooms and one living room is 100 square meters.

[0072] It should be noted that step S10 includes:

[0073] Step A1, receiving the identified object repair instruction and the repaired image of the object to be identified;

[0074] Step A2: If the identified object repair instruction is a misidentification self-learning repair instruction, similarity features are extracted from the repair image of the object to be identified according to the misidentification self-learning repair instruction to obtain the first feature, and the first feature is stored in a preset feature database.

[0075] Specifically, the self-mobile device can capture images or videos through a camera, perform object recognition on the captured images or videos using an AI recognition algorithm, and use each recognized object as an object to be identified and repaired. At the same time, an object frame is drawn for the recognized object to be identified and repaired in the image or video captured by the camera, and the content within the object frame in the image or video is captured to form an image including the object to be identified and repaired, which serves as a repaired image of the object to be identified. The self-mobile device transmits the repaired image of the object to be identified to an application and displays the repaired image of the object to be identified at a corresponding location on a map. Specifically, the display location of the repaired image of the object to be identified can be determined based on the coordinate information of the object frame, that is, the repaired image of the object to be identified of a corresponding size is displayed as a display icon at the actual location of the object on the map for the user to view.

[0076] Furthermore, if the user determines that a misidentification has occurred after viewing the repaired image of the object to be identified and comparing it with the actual scene, the misidentification self-learning repair button is triggered by the application (for example, clicking a button on the screen or a physical button on the smart terminal), and the application generates and sends a misidentification self-learning repair instruction based on the repaired image of the object to be identified to the self-mobile device, thereby sending the repaired image of the object to be identified to the self-mobile device.

[0077] The mobile device receives a misidentification self-learning repair instruction sent by the user through an application, parses the misidentification self-learning repair instruction to generate a repaired image of the object to be identified, and transmits the repaired image of the object to be identified to the AI ​​program via the main program. The AI ​​program inputs the repaired image of the object to be identified into a pre-trained feature extraction network. The pre-trained feature extraction network is run to extract features from the repaired image of the object to be identified, and outputs the misidentified object features of the repaired image of the object to be identified, which can be specifically a 128-dimensional feature vector. The misidentified object features are stored as the first feature in a preset feature database. Furthermore, the user can correct the repaired image of the object to be identified corresponding to the misidentified object by taking a corrected image of the object, which is then used as a corrected image of the repaired image of the object to be identified and stored in association with the repaired object to be identified. For example, if an image of a boot is displayed on a map after being captured and identified by the mobile device, and the user determines after comparison that it is actually a pair of stockings, the user can take an image of the stockings and upload it to the application. The corrected image of the stockings is used as a corrected image of the original boot image and is stored in the preset feature database in association with the misidentified boot features.

[0078] And, step S10 further includes:

[0079] Step B1: if a missed object repair instruction is received, obtaining the missed image and the category of the missed object in the missed object repair instruction;

[0080] Step B2: extracting features from the missed-recognition image based on a pre-trained feature extraction network to obtain features of the missed-recognition object, and storing the features of the missed-recognition object as the first feature in association with the category of the missed-recognition object in a preset feature database.

[0081] It is understood that after the mobile device has cleaned the designated area and displayed the corresponding object image on the map as a display icon, if the user finds that an object has been missed through actual comparison, the user can control the mobile device through the application to take a photo of the area where the object is located. After taking the photo, the user can frame the object in the image preview interface by dragging a frame and capture the content within the frame to form a missed image. Furthermore, the user can select the category of the object in the application and click the Missing Object Repair button after the selection is completed. The application will generate a Missing Object Repair instruction based on the missed image and the category of the missed object and send it to the mobile device. The categories of missed objects include but are not limited to shoes, socks, wires, scales, feces, fan bases, human legs, wallets, carpets, etc.

[0082] A missing object repair instruction sent by a user via an application on a mobile device is received, the missing object repair instruction is parsed, and the missing image and the category of the missing object are parsed from the missing object repair instruction. Furthermore, the missing image is transmitted to the AI ​​program via the main program. The AI ​​program inputs the missing image into a pre-trained feature extraction network, runs the pre-trained feature extraction network, extracts features from the missing image, and outputs features of the missing object as missing object features, specifically a 128-dimensional feature vector. The missing object feature is stored as a first feature in a preset feature database along with the category of the missing object.

[0083] It can be understood that the preset feature database may include multiple first features, and the first feature may be a feature of an erroneously identified object or a feature of a missed object.

[0084] It should be noted that, before extracting features from the restored image of the object to be identified based on the pre-trained feature extraction network to obtain the first feature, the following steps are further included:

[0085] Step C1, obtaining an object image and its annotation information as a training sample;

[0086] Step C2, constructing an initial feature extraction network;

[0087] Step C3: training the initial feature extraction network with a triplet loss function according to the training samples to obtain a pre-trained feature extraction network.

[0088] The mobile device needs to construct an initial feature extraction network and perform pre-training before it can extract features from the inpainted image of the object to be identified using the pre-trained feature extraction network. Specifically, an object image and its annotation information are first obtained as training samples. This can be obtained by capturing an object image and its annotation information through the mobile device's camera and annotating it via an application, or by capturing an object image and its annotation information from a browser taken by someone else. For example, if an object image captured by a camera contains a wallet, and a user annotates the object in the image through an application as a wallet, the resulting image contains an object image and its annotation information. It should be noted that each image and its annotation information is considered a sample, and training samples are formed by acquiring a large number of samples, such as 10,000 sets of samples, 100,000 sets of samples, or 100,000,000 sets of samples.

[0089] At the same time, the mobile device builds an initial feature extraction network based on the current scene. Specifically, the network structure of EfficiencyNet-Lite0 is used. Furthermore, in this embodiment, the loss function uses a triplet loss function. The goal of triplet loss optimization is to ensure that the embedding distances of the same object are as close as possible and the embedding distances between different objects are as far as possible. Embedding is a term from topology and is often used in conjunction with manifold in deep learning. Several examples can illustrate this. For example, a sphere in three-dimensional space is a two-dimensional manifold embedded in three-dimensional space (2D manifold embedded in 3D space). It is called a two-dimensional manifold because any point on the sphere can be expressed using only two-dimensional longitude and latitude. For example, a rotation matrix in two-dimensional space is a 2x2 matrix, but it can actually be expressed using only a single angle. This is a one-dimensional manifold embedded in a 2x2 matrix space. More specifically, we select an image as the anchor. Positive is the same object as the anchor, and negative is a different object from the anchor. We hope to make the anchor closer to the positive and farther from the negative through learning. Therefore, we directly call the combination of these three images a triplet. Then the loss can be defined as follows:

[0090]

[0091] in, The feature vector representing sample i is 128-dimensional, such as the 128-dimensional feature vector of a shoe image; The feature vector that represents the same identity as the sample i vector is 128-dimensional. For example, it is the same shoe as above, but photographed from a different angle. Indicates that the feature vector of sample i is not the same as the feature vector of the same identity, 128 dimensions, for example, the other shoes are not the same as the shoes above; ||*|| is the Euclidean distance; α means that there is a minimum interval between the Euclidean distance between x_a and x_n and the Euclidean distance between x_a and x_p, that is, the offset, which is to prevent the loss from becoming too small to 0, resulting in the gradient being 0, to achieve smoothing, specifically a learnable hyperparameter; + means that if the value in [] is greater than 0, the value is removed as the loss, and if it is less than 0, the value is taken as 0; i = 0, 1, 2..., N.

[0092] Therefore, after the mobile device obtains the training samples and constructs the initial feature extraction network, it can use a large number of samples to train the initial feature extraction network with the triplet loss function as the loss function, so that the prediction effect of the trained feature extraction network is optimized, and a pre-trained feature extraction network is obtained, so that the features contained in the image can be accurately extracted from the input image through the pre-trained feature extraction network.

[0093] Step S20 , when the mobile device detects that the object to be identified and repaired exists in the initial image obtained by photographing the area to be cleaned, similarity features are extracted from the first image of the object to be identified and repaired to obtain a second feature.

[0094] After the map is generated and displayed on the mobile device, when the user has a cleaning need, he can use the application to select the area to be cleaned in the displayed map, such as cleaning three rooms or all areas of the map, etc. The application generates cleaning instructions based on the user's operation and sends them to the mobile device.

[0095] After receiving a cleaning instruction, the self-mobile device uses the user-selected area included in the cleaning instruction as the area to be cleaned and cleans the area to be cleaned. In this embodiment, when the self-mobile device is cleaning, it can detect objects in the uncleaned area and determine whether the area containing the object needs to be cleaned. For example, carpets can be cleaned in boost mode, while shoes, socks, etc. need to be bypassed and not cleaned. During the cleaning process, if the AI ​​(artificial intelligence) recognition algorithm recognizes the initial image or video captured by the camera and determines that any object exists in the area to be cleaned, the object is identified as the first object. Then, an object frame can be drawn for the detected first object in the image captured by the camera, and the content within the object frame in the image can be captured to form an image including the first object as the first image. This facilitates subsequent feature extraction of the first image and a similarity comparison between the extracted features and the features stored in the preset feature database. Based on the similarity comparison results, corresponding processing is performed to avoid misidentification and missed identification, thereby improving the cleaning accuracy of the self-mobile device. The preset feature database is a database used to store feature data. The database is a "warehouse that organizes, stores, and manages data according to a data structure." It is a collection of large amounts of data that is stored in a computer for a long time, organized, shareable, and uniformly managed.

[0096] After generating a first image including a first object, an object recognition repair instruction is received, and the first image is input from the mobile device into a feature extraction network configured internally and pre-trained. Based on the pre-trained feature extraction network and the object recognition repair instruction, feature extraction is performed on the input repaired image of the object to be identified. After the pre-trained feature extraction network completes the feature extraction of the repaired image of the object to be identified, a feature vector of a preset dimension corresponding to the repaired image of the object to be identified is output as a second feature, wherein the preset dimension may be 64 dimensions, 128 dimensions, 256 dimensions, etc., and in this embodiment, 128 dimensions may be preferred. In this embodiment, the feature extraction network may be specifically a CNN. Since it needs to be deployed on the terminal, the CNN in this embodiment may be more specifically a MobileNet, shuffleNet, efficienet-lite series, etc., and preferably efficienet-lite0 of the efficienet-lite series. Among them, MobileNet, shuffleNet, and efficienet-lite are all currently existing neural network types, and this embodiment will not be elaborated in detail here.

[0097] Step S30: When the similarity between the first feature and the second feature is greater than or equal to a preset similarity threshold, updating the first image.

[0098] In step S30, the self-mobile device compares the second feature with each first feature stored in the preset feature database to determine whether there is a second feature in the preset feature database with a similarity greater than a preset similarity threshold. The preset similarity threshold can be set based on actual needs, such as 0.7, 0.8, 0.9, etc. This allows for more accurate recognition of objects captured by the camera, avoids misidentification and missed identification, and improves the cleaning accuracy of the self-mobile device.

[0099] Furthermore, in this embodiment, the first feature and the second feature both include feature vectors of a preset dimension, for example, 128 dimensions. The step of comparing the second feature with the first feature pre-stored in the preset feature database for similarity includes:

[0100] Step S31 : performing similarity comparison between the feature vectors of the preset dimensions in the second feature and the feature vectors of the preset dimensions of the first feature pre-stored in the preset feature database, and determining the similarity between the first feature and the second feature.

[0101] When comparing the second feature with the first feature pre-stored in the preset feature database for similarity, the feature vector of the preset dimension in the second feature, for example, 128 dimensions, can be specifically compared with the 128-dimensional feature vector of each first feature in the preset feature database for similarity. More specifically, the similarity comparison of the feature vectors can be achieved through Euclidean distance or cosine distance, that is, the Euclidean distance between the 128-dimensional feature vector in the first feature and the 128-dimensional feature vector in the second feature is calculated respectively to obtain the similarity between the first feature and each second feature; or, the cosine distance between the 128-dimensional feature vector in the first feature and the 128-dimensional feature vector in the second feature is calculated respectively to obtain the similarity between the first feature and each second feature.

[0102] After comparing the similarity between the first feature and the second feature pre-stored in the preset feature database, if it is determined that there is a first feature in the preset feature database whose similarity with the second feature is greater than the preset similarity threshold, for example, there is a second feature in the preset feature database whose similarity with the first feature is greater than 0.7, it means that the self-mobile device has made misidentification or missed identification in the previous cleaning and identification process; therefore, when the first feature whose similarity with the second feature is greater than the preset similarity threshold is a missed object feature, the category of the missed object corresponding to the first feature is determined as the category of the first object, and the cleaning method of the area to be cleaned is determined according to the category of the first object, where the cleaning method in this embodiment is specifically cleaning or bypassing without cleaning. Therefore, specifically, it is determined whether the area where the object is located needs to be cleaned according to the category of the first object. If it is necessary, it is cleaned; if it is not necessary, the area is bypassed to clean the rest of the area to be cleaned. When the first feature, whose similarity to the second feature is greater than a preset similarity threshold, is a feature of a misidentified object, the second image is replaced with the correct image of the misidentified object corresponding to the first feature, and the cleaning method of the area to be cleaned is determined based on the correct image. Specifically, the correct image is used to determine whether the area where the object corresponding to the image is located needs to be cleaned. If so, it is cleaned; if not, the area is bypassed to clean the remaining areas in the area to be cleaned. When determining whether the area where the object is located needs to be cleaned based on the category of the first object or determining whether the area where the object corresponding to the image is located needs to be cleaned based on the correct image, if the category of the first object or the object in the correct image is a carpet, etc., and cleaning will not affect the object or the self-moving device, then the area where the object is located needs to be cleaned, for example, the entire carpet area. If the category of the first object or the object in the correct image is a sock, shoe, or other object that may affect the object or the self-moving device, then the area where the object is located does not need to be cleaned or cannot be cleaned. This avoids misidentification and missed identification, and improves the cleaning accuracy of the self-moving device.

[0103] It can be understood that if the similarities between all first features and second features in the preset feature database are less than or equal to the preset similarity threshold, it is determined that there is no misidentification or missed identification of the detected first object, and it is a normal identification. The first image and the category of the first object in the first image are then transmitted to the application, and the application displays the first image at the corresponding position on the map as a display icon for the object at that position and displays the category of the object at the same time.

[0104] This embodiment provides a self-learning recognition method for a self-mobile device, which receives an identification object repair instruction and a repaired image of an object to be identified, and performs similarity feature extraction on the repaired image of the object to be identified according to the identification object repair instruction to obtain a first feature; wherein, the repaired image of the object to be identified includes the repaired object to be identified; when the self-mobile device detects that the repaired object to be identified exists in an initial image obtained by photographing the area to be cleaned, performs similarity feature extraction on the first image of the repaired object to be identified to obtain a second feature; wherein, the initial image includes the first image; when the similarity between the first feature and the second feature is greater than or equal to a preset similarity threshold, the first image is updated. In the present application, the self-mobile device can generate an initial image with the identification of the object when an object is detected during the cleaning process. After feature extraction of the generated initial image to obtain a second feature, the similarity is compared with the first feature stored in a preset feature database. When the similarity between the first feature and the second feature is greater than a preset similarity threshold, the category of the missed object corresponding to the first feature is determined as the category of the object, or the correct image of the misidentified object corresponding to the first feature is used to replace the first image. In this way, the objects in the area to be cleaned are first accurately identified and then cleaned according to the correct image or category of the object, avoiding misidentification and missed identification, and improving the cleaning accuracy of the self-mobile device.

[0105] Furthermore, based on the first embodiment of the self-learning recognition method for a mobile device of the present application, a second embodiment of the self-learning recognition method for a mobile device of the present application is proposed. In the second embodiment, the second feature includes a missed object feature, and the step of determining the category of the missed object corresponding to the second feature as the category of the first object includes:

[0106] Step S411: if the second feature is a missed object feature, obtaining a category of the missed object corresponding to the missed object feature;

[0107] Step S412: display the first image, and determine the object category associated with the second feature as the category of the object to be identified and repaired.

[0108] After determining that the similarity between the first feature and the second feature is greater than a preset similarity threshold, if the first feature is a missed object feature and it is determined that the currently detected obstacle has been previously identified as a missed object by the user, the object category associated with the missed object feature is obtained as the missed object category, and the missed object category is determined as the category of the currently detected first object, so as to determine whether the corresponding area needs to be cleaned based on the category of the first object. For example, if the similarity between the currently detected feature of a sock and the feature of a sock pre-stored in the preset feature data is greater than 0.7, indicating that the same or similar socks have been missed before, the category set by the user for the missed sock when confirming missed recognition is determined as the category of the object, i.e., socks.

[0109] Furthermore, the first feature includes a feature of a misidentified object, and the step of replacing the first image with a correct image corresponding to the misidentified object according to the first feature includes:

[0110] Step S421: if the second feature is a misidentified object feature, obtaining a correct image of the misidentified object corresponding to the misidentified object feature;

[0111] Step S422: Replace the first image with the object image associated with the second feature.

[0112] After determining that the similarity between the first feature and the second feature is greater than a preset similarity threshold, if the first feature is a misidentified object feature and it is determined that the currently detected first object has been previously identified as a misidentified object by the user, a correct image associated with the misidentified object feature corresponding to the misidentified object feature is obtained, the first image is replaced with the correct image, and the correct image is transmitted to the self-mobile device for display. For example, if there is a boot in the area to be cleaned, but the self-mobile device mistakenly identifies it as a stocking after identification, the self-mobile device compares the extracted features with the features of a boot that was also previously mistakenly identified as a stocking, and determines that the similarity between the extracted features and the features of the misidentified boot is greater than 0.7, then a correct image of the boot taken by the user is obtained, and the first image is replaced with the correct image of the boot.

[0113] It is understood that after determining that the similarity between the first feature and the second feature is greater than a preset similarity threshold, if the first feature is a misidentified object feature, this embodiment can also directly filter out the first image, that is, not display the display icon of the misidentified image on the map. For example, if a pair of stockings is misidentified as a boot, the originally generated image is filtered out and not displayed on the map.

[0114] After determining that the similarity between the first feature and the second feature is greater than a preset similarity threshold, this embodiment can determine whether the area corresponding to the first object needs to be cleaned based on the corresponding processing method of misidentification or missed identification, thereby avoiding misidentification and missed identification and improving the cleaning accuracy of the self-moving device.

[0115] In a specific embodiment of the present application, refer to Figure 3 , Figure 3 This is a flowchart of a specific embodiment of the self-learning recognition method for a mobile device of the present application. In this embodiment, the mobile device is a sweeper, and the application program is called an app. After the sweeper detects an obstacle and identifies it as a shoe, it uploads the image of the shoe to the app for display on a map. After the app displays the shoe image on the map, the user determines whether it is a misidentified shoe. If not, no action is taken. If it is, the user clicks the "misidentification self-learning repair" button on the app, and the app sends the misidentified first image as a thumbnail to the sweeper. After receiving the thumbnail, the sweeper transmits it to the AI ​​program through the main program. The AI ​​program runs a pre-trained similarity feature extraction network, outputs a 128-dimensional feature, and stores the feature in a local misidentification feature database. At the beginning of the next recognition, that is, the next round of cleaning, if an obstacle is detected and a thumbnail is generated based on the obstacle, the thumbnail is input into the similarity feature extraction network, which then runs to output a 128-dimensional feature. The feature is compared with the local misidentification feature database for similarity to determine whether the similarity meets a preset similarity threshold. If the similarity between this feature and a certain feature is large, for example, greater than 0.7, the small image will be filtered out and not output or displayed on the app's map; if the similarity between this feature and all features in the local misidentification feature library is small, for example, less than or equal to 0.7, the normal recognition process will be carried out, that is, it is considered that there is no misidentification or missed recognition, and the small image will be output to the app's map for display.

[0116] In another specific embodiment of the present application, refer to Figure 4 , Figure 4This is a flow chart of another specific embodiment of the self-learning recognition method of the self-mobile device of the present application; in this embodiment, the self-mobile device is a sweeping machine, and the application program is called an app. Specifically, since objects with a fixed appearance are often not recognized, the user can control the machine, i.e., the sweeping machine, to take pictures through the app, and in the photo taking interface, the user can draw a frame to frame the object, and after framing the object, select the category of the object, and click the missed object repair button, and the picture of the object in the frame is sent to the sweeping machine as a thumbnail and category information. After receiving the thumbnail, the sweeping machine transmits the thumbnail to the AI ​​program through the main program. The AI ​​program runs the pre-trained similarity feature extraction network, outputs a 128-dimensional feature, and stores the feature in the local missed recognition feature database. At the beginning of the next recognition, that is, at the beginning of the next round of cleaning, if an obstacle is detected and a thumbnail is obtained based on the obstacle, the thumbnail is input into the similarity feature extraction network, and the 128-dimensional feature is output by running the similarity feature extraction network. The feature is compared with the local missed recognition feature library for similarity to determine whether the similarity meets the preset similarity threshold. If the similarity between the feature and a certain feature is large, for example, greater than 0.7, the category and small picture corresponding to the feature are output; if the similarity between the feature and all features in the local misidentification feature library is small, for example, less than or equal to 0.7, the normal recognition process is carried out, that is, it is considered that there is no misidentification or missed recognition, and the small picture is output to the map of the app for display.

[0117] Furthermore, the present application also provides a cleaning device for a self-moving device.

[0118] Reference Figure 5 , Figure 5 This is a schematic diagram of the functional modules of the first embodiment of the cleaning device for mobile equipment of the present application.

[0119] The cleaning device of the self-moving device includes:

[0120] The first feature extraction module 10 is used to receive an identification object repair instruction and a repaired image of the object to be identified, and extract similar features from the repaired image of the object to be identified according to the identification object repair instruction to obtain a first feature; wherein the repaired image of the object to be identified includes the repaired object to be identified.

[0121] The second feature extraction module 20 is used to extract similar features from the first image of the object to be identified and repaired to obtain a second feature when the mobile device detects that the object to be identified and repaired exists in the initial image obtained by photographing the area to be cleaned; wherein the initial image includes the first image.

[0122] The updating module 30 is configured to update the first image when the similarity between the first feature and the second feature is greater than or equal to a preset similarity threshold.

[0123] In addition, the present application also provides a medium, which is preferably a computer-readable storage medium, on which a cleaning program for a self-mobile device is stored. When the cleaning program for a self-mobile device is executed by a processor, the steps of each embodiment of the self-learning identification method of the self-mobile device are implemented.

[0124] In the embodiments of the self-mobile device and computer-readable storage medium of the present application, all technical features of the above-mentioned embodiments of the self-learning identification method of the self-mobile device are included, and the description and explanation content are basically the same as those of the above-mentioned embodiments of the self-learning identification method of the self-mobile device, and will not be repeated here.

[0125] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0126] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for enabling a terminal device (which can be a fixed terminal, such as an IoT smart device, including smart air conditioners, smart lights, smart power supplies, smart routers and other smart homes; or a mobile terminal, including smart phones, wearable networked AR / VR devices, smart speakers, self-driving cars and many other networked devices) to execute the methods described in the various embodiments of the present application.

[0128] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A self-learning recognition method for a mobile device, characterized in that: The self-learning identification method of the mobile device includes: When the mobile device detects an obstacle, the image of the obstacle is uploaded to the application as a repaired image of the object to be identified and displayed; receiving a misidentification self-learning repair instruction and a repair image of an object to be identified triggered by a user on the application; Extracting similar features from the repaired image of the object to be identified according to the false recognition self-learning repair instruction to obtain a first feature, and storing the first feature in a false recognition feature database; wherein the repaired image of the object to be identified includes the object to be identified and repaired; When the mobile device detects that the object to be identified and repaired exists in an initial image obtained by photographing the area to be cleaned, similarity feature extraction is performed on the image of the object to be identified and repaired to obtain a second feature; The second feature is compared with the first feature in the misidentification feature database for similarity to obtain a comparison result, and based on the comparison result, it is determined whether the image of the object to be identified and repaired is displayed in the map of the application, and based on the comparison result, it is determined whether to replace the image of the object to be identified and repaired with the image of the object corresponding to the second feature.

2. The self-learning recognition method for a mobile device according to claim 1, wherein: The step of determining whether to replace the object to be identified and repaired according to the object image corresponding to the second feature based on the comparison result includes: When the similarity between the first feature and the second feature is greater than or equal to a preset similarity threshold, the image of the object to be identified and repaired is replaced according to the image of the object corresponding to the second feature.

3. The self-learning recognition method for a mobile device according to claim 2, wherein: The step of determining whether the image of the object to be identified and repaired is displayed in the map of the application based on the comparison result includes: When the similarity between the first feature and the second feature is greater than or equal to a preset similarity threshold, the image of the object to be identified and repaired is displayed, and the object category associated with the second feature is determined as the category of the object to be identified and repaired.

4. The self-learning recognition method for a mobile device according to claim 1, wherein: After the step of uploading the image of the obstacle as the repaired image of the object to be identified to the application and displaying the image when the mobile device detects the obstacle, the following steps are performed: receiving the identified object repair instruction and the repaired image of the object to be identified; If the identified object repair instruction is a missed object repair instruction, performing similarity feature extraction on the repaired image of the object to be identified according to the missed object repair instruction to obtain the first feature, and acquiring category information of the first feature; The first feature and the category information of the first feature are associated and stored in the misidentification feature database.

5. The self-learning recognition method for a mobile device according to claim 1, wherein: The comparing the second feature with the first feature in the misidentification feature database for similarity comprises: The feature vectors of the preset dimensions in the second feature are respectively compared with the feature vectors of the preset dimensions of the first feature pre-stored in the misidentification feature database to determine the similarity between the first feature and the second feature.

6. The self-learning identification method for a mobile device according to claim 1, wherein: Before the step of uploading the image of the obstacle as a repaired image of the object to be identified to the application and displaying the image when the mobile device detects the obstacle, the method further includes: Obtain object images and their annotation information as training samples; Build the initial feature extraction network; Training the initial feature extraction network using a triplet loss function according to the training samples to obtain a pre-trained feature extraction network; The extracting of similar features from the repaired image of the object to be identified according to the misidentification self-learning repair instruction to obtain the first feature includes: Based on the pre-trained feature extraction network, similar features are extracted from the repaired image of the object to be identified according to the identification object repair instruction to obtain the first feature.

7. A cleaning device for a self-propelled device, characterized in that: The cleaning device of the self-moving device includes: A first feature extraction module is configured to, when the self-mobile device detects an obstacle, upload an image of the obstacle as a repaired image of the object to be identified to an application and display the image; receive a misidentification self-learning repair instruction and a repaired image of the object to be identified triggered by a user on the application; extract similar features from the repaired image of the object to be identified based on the misidentification self-learning repair instruction to obtain the first feature, and store the first feature in a misidentification feature database; wherein the repaired image of the object to be identified includes the repaired object to be identified; A second feature extraction module is configured to extract similar features from the image of the object to be identified and repaired to obtain a second feature when the mobile device detects that the object to be identified and repaired exists in the initial image obtained by photographing the area to be cleaned; An updating module is used to compare the second feature with the first feature in the misidentification feature database for similarity to obtain a comparison result, determine whether the image of the object to be identified and repaired is displayed in the map of the application based on the comparison result, and determine whether to replace the image of the object to be identified and repaired with the image of the object corresponding to the second feature based on the comparison result.

8. A self-propelled device, characterized in that: The self-mobile device includes a memory, a processor, and a cleaning program of the self-mobile device stored in the memory and executable on the processor. When the cleaning program of the self-mobile device is executed by the processor, the steps of the self-learning identification method of the self-mobile device as described in any one of claims 1-6 are implemented.

9. A medium, which is a computer-readable storage medium, characterized in that: The computer-readable storage medium stores a cleaning program for the self-mobile device, and when the cleaning program for the self-mobile device is executed by the processor, the steps of the self-learning identification method for the self-mobile device as described in any one of claims 1 to 6 are implemented.

Images