A robot scene recognition method, system, storage medium and terminal based on semantic grid map

Abstract

The present invention discloses a robot scene recognition method, system, storage medium, and terminal based on a semantic grid map. This method belongs to the field of indoor mobile robot scene recognition technology and includes the following steps: obtaining a semantic grid map and obtaining a closed contour map based on the semantic grid map; performing region segmentation and semantic object category extraction on the closed contour map to obtain subregions and semantic objects; calculating the probability that the current subregion is a specific home scene, and identifying the scene in which the subregion resides based on the probability. The present invention's scene recognition method based on a semantic grid map is not limited by models or computing power and can enable a sweeping robot to quickly and accurately identify scenes, effectively completing scene-related command tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of indoor mobile robot scene recognition, and in particular to a robot scene recognition method, system, storage medium and terminal based on semantic grid map. Background Art

[0002] With the progress of society and the development of technology, people are pursuing a higher quality of life. They are increasingly seeking robot vacuums that can intelligently adapt cleaning modes and intensity to different room types. Traditional robot vacuums, which only offer basic modules like cleaning and obstacle avoidance, are gradually being phased out as they fail to meet people's needs. Therefore, the application of scene recognition algorithms in the field of robot vacuums has become a mainstream trend.

[0003] Current scene recognition algorithms are primarily based on deep learning. By extracting visual features from images and training scene classifiers, they achieve final scene recognition results. Images collected in different environments, as well as the selection and configuration of different scene classifiers, significantly impact the scene recognition algorithm. This deep learning-based scene recognition approach is significantly limited by the images collected and the classification model selected. Furthermore, the model is large and requires high computing power. Therefore, applying deep learning-based scene recognition algorithms to sweeping robots is currently difficult, and scene recognition performance is poor in diverse home environments. Summary of the Invention

[0004] The purpose of the present invention is to overcome the problems existing in scene recognition of existing indoor mobile robots, and to provide a robot scene recognition method, system, storage medium and terminal based on semantic grid map.

[0005] The object of the present invention is achieved through the following technical solutions:

[0006] In a first aspect, a method for robot scene recognition based on a semantic grid map is provided, the method comprising the following steps:

[0007] S1. Obtain a semantic grid map, and obtain a closed contour map based on the semantic grid map;

[0008] S2. Perform region segmentation and semantic object category extraction on the closed contour map to obtain sub-regions and semantic objects;

[0009] S3. Calculate the probability value that the current sub-area is a specific home scene, and identify the scene where the sub-area is located based on the probability value.

[0010] In some possible embodiments, a robot scene recognition method based on a semantic grid map is provided, wherein step S1 includes:

[0011] Separate semantic raster map outlines from map objects;

[0012] Perform rotation correction on the separated overall contour image;

[0013] The corrected contour map is then region-enclosed.

[0014] In some possible embodiments, a robot scene recognition method based on a semantic grid map is provided, wherein step S2 includes:

[0015] The connected domain of the closed contour map is calculated, and the semantic grid map is segmented and extracted based on the coordinates, width and height information of the circumscribed rectangle of each connected domain to obtain sub-regions similar to the room;

[0016] Connected domains are calculated for the sub-regions. Each connected domain obtained is an object. The color of the object is extracted, and whether the object has semantics is determined based on the color. If it is a semantic object, the object category corresponding to the color is queried.

[0017] In some possible embodiments, a robot scene recognition method based on a semantic grid map is provided, wherein step S3 includes:

[0018] Calculate the probability of M types of iconic objects in N types of specific home scenes;

[0019] Calculate the probability that the semantic objects in the sub-region belong to a certain type of specific home scene;

[0020] According to the scene recognition probability, the specific home scene where the sub-area is located is determined, and the corresponding color is attached to the sub-area.

[0021] In some possible embodiments, a method for robot scene recognition based on a semantic grid map is provided, wherein the method of calculating the probability of M types of landmark objects in N types of specific home scenes includes:

[0022] Let each specific home scene be ξ i (i=1,…N), each iconic object is denoted as O j (j=1,…M), the prior probability of the iconic object appearing in each specific home scene is p(O j＝1 …M|ξ i＝1 …N).

[0023] In some possible embodiments, a method for robot scene recognition based on a semantic grid map is provided, wherein the method of calculating the probability that a semantic object in a sub-region belongs to a certain type of specific home scene includes:

[0024] When only one object O is detected in the image t When the picture scene is a specific home scene s The probability is:

[0025]

[0026] When k kinds of iconic objects are detected in the image, the image scene is a specific home scene ξ s The probability is:

[0027]

[0028] In some possible embodiments, a robot scene recognition method based on a semantic grid map is provided, wherein determining the specific home scene in which the sub-area is located based on the scene recognition probability includes:

[0029] The scene corresponding to the maximum probability value is selected as the scene where the sub-region is located.

[0030] In a second aspect, a robot scene recognition system based on a semantic grid map is provided, comprising:

[0031] a closed contour map making module configured to obtain a semantic grid map and obtain a closed contour map based on the semantic grid map;

[0032] A region segmentation and semantic object extraction module is configured to perform region segmentation and semantic object category extraction on the closed contour image to obtain subregions and semantic objects;

[0033] The scene recognition module is configured to calculate the probability value of the current sub-area being a specific home scene, and identify the scene where the sub-area is located based on the probability value.

[0034] In a third aspect, a computer storage medium is provided, on which computer instructions are stored, and when the computer instructions are executed, the relevant steps in any one of the robot scene recognition methods based on semantic grid maps are executed.

[0035] In a fourth aspect, a terminal is provided, comprising a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, and when the processor executes the computer instructions, the processor executes any relevant step in the robot scene recognition method based on a semantic grid map.

[0036] It should be further explained that the technical features corresponding to the above options can be combined or replaced with each other to form a new technical solution if there is no conflict.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention's robot scene recognition method, based on semantic grid maps, segments the acquired semantic grid map into regions. Based on the semantic objects in the segmented subregions, the naive Bayesian method is used to calculate scene recognition probability values, obtaining the final scene recognition results and marking them on the corresponding grid map. Compared to scene recognition algorithms based on deep learning, this method is unaffected by environmental changes, classification model selection, and computing power. It exhibits a certain degree of stability, can quickly and accurately complete scene recognition, and can better complete scene-related command tasks. Furthermore, the present invention's overall framework model is relatively clear and concise, does not require a large network structure and parameters, is easy to transplant, and has good real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of a robot scene recognition method based on a semantic grid map according to an embodiment of the present invention;

[0040] Figure 2 A flow chart of obtaining a closed contour graph according to an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram illustrating the region closing of a corrected contour image according to an embodiment of the present invention;

[0042] Figure 4 This is a flowchart of scene recognition according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical solution of the present invention is described clearly and completely below with reference to the accompanying drawings. It is apparent that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0044] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0045] Reference Figure 1 In an exemplary embodiment, a robot scene recognition method based on a semantic grid map is provided, the method comprising the following steps:

[0046] S1. Obtain a semantic grid map, and obtain a closed contour map based on the semantic grid map;

[0047] S2. Perform region segmentation and semantic object category extraction on the closed contour map to obtain sub-regions and semantic objects;

[0048] S3. Calculate the probability value that the current sub-area is a specific home scene, and identify the scene where the sub-area is located based on the probability value.

[0049] Further, refer to Figure 2 , the step S1 comprises:

[0050] Separate the semantic raster map outline from the objects within the map. Specifically, the map outline is separated from the semantic objects within the map based on color information. In a semantic raster map, the map outline is black, and semantic objects are colored. The map outline is separated from the semantic objects based on whether the color is black. Then, the map outline is separated from the non-semantic black objects within the map based on the area of ​​the connected regions. Non-semantic black objects have a small connected region area, so they can be removed by determining whether the connected region area is less than a specific threshold.

[0051] The separated overall contour image is rotated and corrected. Specifically, the row operator norm of the contour image at different angles is calculated. The angle corresponding to the maximum row operator norm is the angle by which the contour image needs to be rotated and corrected. The row operator norm is used to calculate and determine the position of walls in the map. The angle corresponding to the maximum row operator norm is the angle by which the entire map needs to be rotated and corrected to obtain a correct top-down view of the entire map.

[0052] The corrected contour image is closed. Specifically, the number of black dots in each row and column is calculated to see if it is greater than the threshold 1 to determine whether the row or column needs to be connected to close the area. If the number of black dots in the row or column is greater than the threshold 1, it means that the row or column may be a wall and the gap needs to be connected. If there is a gap in the black line and the gap length is less than the threshold 2, it means that the gap is a door. Add a black dot to the gap to complete the area closure. The operation is shown in the figure below. Figure 3 shown.

[0053] Furthermore, the step S2 includes:

[0054] The connected domain of the closed contour map is calculated, and the semantic grid map is segmented and extracted based on the coordinates, width and height information of the circumscribed rectangle of each connected domain to obtain sub-regions similar to the room;

[0055] Connected domains are calculated for the sub-regions. Each connected domain obtained is an object. The color of the object is extracted, and whether the object has semantics is determined based on the color. If it is a semantic object, the object category corresponding to the color is queried.

[0056] Furthermore, the naive Bayes method is used to realize scene recognition based on object categories, referring to Figure 4 , the step S3 comprises:

[0057] Calculate the probability of M types of iconic objects in N types of specific home scenes. Specifically, collect indoor home scene datasets and count the probabilities of M types of iconic objects in N types of specific home scenes. Iconic objects can be beds, refrigerators, sofas, toilets, dining tables, coffee tables, etc. Let each specific home scene be denoted by ξ i (i=1,…N), each iconic object is denoted as O j (j=1,…M), the prior probability of the iconic object appearing in each specific home scene is p(O j＝1 …M|ξ i＝1 …N).

[0058] Calculate the probability that the semantic object in the sub-region belongs to a certain type of home scene. Specifically, when only one object O is detected in the image t When the picture scene is a specific home scene s The probability is:

[0059]

[0060] When k kinds of iconic objects are detected in the image, the image scene is a specific home scene ξ s The probability is:

[0061]

[0062] The core idea of ​​this computational method is Naive Bayes, which is based on the assumption that whether each iconic object appears in a scene is independent of each other.

[0063] According to the scene recognition probability, the specific home scene where the sub-area is located is determined, and the corresponding color is attached to the sub-area. Among them, the scene corresponding to the maximum probability value is selected as the scene where the sub-area is located, and the color corresponding to the specific home scene is attached to this area.

[0064] In another exemplary embodiment, a robot scene recognition system based on a semantic grid map is provided, comprising:

[0065] a closed contour map making module configured to obtain a semantic grid map and obtain a closed contour map based on the semantic grid map;

[0066] A region segmentation and semantic object extraction module is configured to perform region segmentation and semantic object category extraction on the closed contour image to obtain subregions and semantic objects;

[0067] The scene recognition module is configured to calculate the probability value of the current sub-area being a specific home scene, and identify the scene where the sub-area is located based on the probability value.

[0068] In another exemplary embodiment, the present invention provides a computer storage medium having computer instructions stored thereon, which, when executed, execute relevant steps in the robot scene recognition method based on semantic grid map.

[0069] Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0070] In another exemplary embodiment, the present invention provides a terminal including a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, and when the processor executes the computer instructions, the relevant steps in the robot scene recognition method based on the semantic grid map are executed.

[0071] The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0072] Embodiments of the subject matter and functional operations described in this specification may be implemented in: tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or to control the operation of the data processing apparatus. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode and transmit information to a suitable receiver apparatus for execution by the data processing apparatus.

[0073] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform the corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0074] Processors suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, a central processing unit will receive instructions and data from a read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to such a mass storage device to receive data from it or to transmit data to it, or both. However, a computer does not necessarily have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0075] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of specific embodiments of specific inventions. Certain features described in multiple embodiments within this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of the sub-combination.

[0076] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0077] The above specific implementation methods are detailed descriptions of the present invention. It cannot be considered that the specific implementation methods of the present invention are limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can make several simple deductions and substitutions without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A robot scene recognition method based on semantic grid map, characterized in that: The method comprises the following steps: S1. Obtain a semantic grid map, and obtain a closed contour map based on the semantic grid map; S2, performing region segmentation and semantic object category extraction on the closed contour map to obtain subregions and semantic objects; said step S2 includes: The connected domain of the closed contour map is calculated, and the semantic grid map is segmented and extracted according to the coordinates, width and height information of the circumscribed rectangle of each connected domain to obtain sub-regions; Calculate the connected domain of the sub-region, and each connected domain obtained is an object. Extract the color of the object, and determine whether the object has semantics based on the color. If it is a semantic object, query the object category corresponding to the color; S3. Calculate the probability value that the current sub-area is a specific home scene, and identify the scene where the sub-area is located based on the probability value.

2. A robot scene recognition method based on semantic grid map according to claim 1, characterized in that: The step S1 comprises: Separate semantic raster map outlines from map objects; Perform rotation correction on the separated overall contour image; The corrected contour map is then region-enclosed.

3. The robot scene recognition method based on semantic grid map according to claim 1, characterized in that: The step S3 comprises: Calculate the probability of M types of iconic objects in N types of specific home scenes; Calculate the probability that the semantic objects in the sub-region belong to a certain type of specific home scene; According to the scene recognition probability, the specific home scene where the sub-area is located is determined, and the corresponding color is attached to the sub-area.

4. The robot scene recognition method based on semantic grid map according to claim 3 is characterized in that: Calculating the probability of M types of iconic objects in N types of specific home scenes includes: Let each specific home scene be , each iconic object is recorded as , the prior probability of the iconic object appearing in each specific home scene is .

5. The robot scene recognition method based on semantic grid map according to claim 4 is characterized in that: The calculation of the probability that a semantic object in a sub-region belongs to a certain type of specific home scene includes: When only one object is detected in the image When the picture scene is a specific home scene The probability is: ; When k iconic objects are detected in the image, the image scene is a specific home scene. The probability is: .

6. The robot scene recognition method based on semantic grid map according to claim 5, characterized in that: The determining of the specific home scene where the sub-area is located based on the scene recognition probability includes: The scene corresponding to the maximum probability value is selected as the scene where the sub-region is located.

7. A robot scene recognition system based on semantic grid map, characterized in that: include: a closed contour map making module configured to obtain a semantic grid map and obtain a closed contour map based on the semantic grid map; The region segmentation and semantic object extraction module is configured to perform region segmentation and semantic object category extraction on the closed contour map to obtain subregions and semantic objects; the region segmentation and semantic object category extraction on the closed contour map to obtain subregions and semantic objects includes: The connected domain of the closed contour map is calculated, and the semantic grid map is segmented and extracted according to the coordinates, width and height information of the circumscribed rectangle of each connected domain to obtain sub-regions; Calculate the connected domain of the sub-region, and each connected domain obtained is an object. Extract the color of the object, and determine whether the object has semantics based on the color. If it is a semantic object, query the object category corresponding to the color; The scene recognition module is configured to calculate the probability value of the current sub-area being a specific home scene, and identify the scene where the sub-area is located based on the probability value.

8. A computer storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed, the relevant steps of the robot scene recognition method based on semantic grid map described in any one of claims 1 to 6 are executed.

9. A terminal comprising a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, characterized in that: When the processor runs the computer instructions, it executes the relevant steps in the robot scene recognition method based on semantic grid map described in any one of claims 1-6.

Images